Jul 02 18:28:20.940447 ip-10-0-140-68 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 02 18:28:20.940456 ip-10-0-140-68 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 02 18:28:20.940463 ip-10-0-140-68 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 02 18:28:20.940679 ip-10-0-140-68 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 02 18:28:31.027266 ip-10-0-140-68 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 02 18:28:31.027285 ip-10-0-140-68 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 851a49ec65de47b3a0b49c96394ece60 --
Jul 02 18:30:07.805546 ip-10-0-140-68 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 18:30:08.215939 ip-10-0-140-68 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 18:30:08.215939 ip-10-0-140-68 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 18:30:08.215939 ip-10-0-140-68 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 18:30:08.215939 ip-10-0-140-68 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 18:30:08.215939 ip-10-0-140-68 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 18:30:08.218875 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.218780    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 18:30:08.222848 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222823    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 18:30:08.222848 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222844    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 18:30:08.222848 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222850    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 18:30:08.222848 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222854    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222859    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222863    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222868    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222873    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222877    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222881    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222884    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222888    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222892    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222896    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222899    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222903    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222907    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222911    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222914    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222918    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222922    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222926    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222929    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 18:30:08.223020 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222932    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222936    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222940    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222945    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222948    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222951    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222955    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222958    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222972    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222976    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222980    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222983    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222987    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222991    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.222996    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223003    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223008    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223013    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223018    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223022    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 18:30:08.223517 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223026    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223030    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223034    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223040    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223044    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223048    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223052    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223056    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223059    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223063    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223067    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223071    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223075    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223079    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223082    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223086    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223089    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223093    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223097    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223101    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 18:30:08.224039 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223104    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223109    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223113    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223117    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223121    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223125    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223130    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223134    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223137    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223142    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223145    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223149    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223152    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223156    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223159    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223163    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223168    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223174    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223178    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 18:30:08.224551 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223184    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223188    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223206    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223210    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223214    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223218    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223222    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223225    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223231    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223237    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223241    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223244    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223248    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223252    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223256    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223260    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 18:30:08.225026 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.223265    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 18:30:08.227075 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227057    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 18:30:08.227115 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227076    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 18:30:08.227115 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227083    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 18:30:08.227115 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227087    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 18:30:08.227115 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227091    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 18:30:08.227115 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227095    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 18:30:08.227115 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227099    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 18:30:08.227115 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227103    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 18:30:08.227115 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227107    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 18:30:08.227115 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227110    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 18:30:08.227115 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227114    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 18:30:08.227115 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227118    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227122    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227128    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227133    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227137    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227141    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227145    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227148    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227152    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227156    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227159    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227163    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227166    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227170    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227173    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227177    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227183    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227187    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227209    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 18:30:08.227403 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227214    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227224    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227228    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227232    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227237    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227241    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227245    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227248    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227252    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227256    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227260    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227264    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227267    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227271    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227274    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227279    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227283    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227288    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227293    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227296    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 18:30:08.227860 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227300    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227304    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227307    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227311    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227315    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227319    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227323    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227326    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227330    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227333    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227337    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227341    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227345    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227348    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227353    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227357    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227361    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227365    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227369    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 18:30:08.228368 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227373    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227376    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227382    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227386    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227390    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227394    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227397    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227401    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227405    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227410    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227413    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227417    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227422    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227425    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227428    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227432    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227435    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227439    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227444    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227447    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 18:30:08.228861 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227451    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227454    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227457    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227461    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227465    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227468    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227472    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227475    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227479    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.227483    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228043    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228049    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228053    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228057    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228061    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228065    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228069    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228073    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228077    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228080    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 18:30:08.229381 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228085    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228089    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228093    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228097    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228102    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228105    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228109    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228113    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228116    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228120    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228124    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228127    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228130    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228134    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228137    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228141    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228145    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228148    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228152    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228157    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 18:30:08.229885 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228164    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228169    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228173    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228176    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228180    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228183    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228187    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228207    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228211    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228215    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228218    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228221    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228226    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228230    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228236    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228240    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228243    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228247    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228250    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228254    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 18:30:08.230398 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228258    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228262    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228266    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228269    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228273    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228277    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228280    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228284    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228287    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228291    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228295    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228298    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228304    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228310    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228315    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228319    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228323    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228327    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228331    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 18:30:08.230880 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228335    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228339    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228343    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228346    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228350    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228354    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228360    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228364    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228367    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228370    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228374    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228377    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228381    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228385    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228390    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228395    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228399    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228403    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228407    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228411    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 18:30:08.231397 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228415    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228418    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228422    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228425    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228429    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228432    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228436    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228441    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228445    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.228449    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228548    2575 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228558    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228574    2575 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228582    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228587    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228592    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228599    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228605    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228611    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228615    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228620    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 18:30:08.231881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228625    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228629    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228634    2575 flags.go:64] FLAG: --cgroup-root=""
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228641    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228646    2575 flags.go:64] FLAG: --client-ca-file=""
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228650    2575 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228654    2575 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228665    2575 flags.go:64] FLAG: --cluster-domain=""
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228670    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228675    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228680    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228684    2575 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228689    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228694    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228698    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228703    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228708    2575 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228712    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228717    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228732    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228737    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228743    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228748    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228752    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228757    2575 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 18:30:08.232425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228761    2575 flags.go:64] FLAG: --enable-server="true"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228764    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228774    2575 flags.go:64] FLAG: --event-burst="100"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228778    2575 flags.go:64] FLAG: --event-qps="50"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228782    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228787    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228791    2575 flags.go:64] FLAG: --eviction-hard=""
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228801    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228805    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228809    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228815    2575 flags.go:64] FLAG: --eviction-soft=""
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228820    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228824    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228829    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228833    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228838    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228842    2575 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228846    2575 flags.go:64] FLAG: --feature-gates=""
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228851    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228855    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228860    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228864    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228868    2575 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228873    2575 flags.go:64] FLAG: --help="false"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228878    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.233119 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228882    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228886    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228898    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228904    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228908    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228913    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228917    2575 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228921    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228926    2575 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228930    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228936    2575 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228940    2575 flags.go:64] FLAG: --kube-reserved=""
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228944    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228948    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228953    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228957    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228961    2575 flags.go:64] FLAG: --lock-file=""
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228967    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228971    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228976    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228983    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228987    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228991    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 18:30:08.233818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.228996    2575 flags.go:64] FLAG: --logging-format="text"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229001    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229006    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229009    2575 flags.go:64] FLAG: --manifest-url=""
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229014    2575 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229020    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229024    2575 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229030    2575 flags.go:64] FLAG: --max-pods="110"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229035    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229039    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229044    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229048    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229061    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229068    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229073    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229084    2575 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229088    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229092    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229097    2575 flags.go:64] FLAG: --pod-cidr=""
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229101    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229109    2575 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229113    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229117    2575 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229122    2575 flags.go:64] FLAG: --port="10250"
Jul 02 18:30:08.234480 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229126    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229130    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eede6284c42c2719"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229141    2575 flags.go:64] FLAG: --qos-reserved=""
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229145    2575 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229150    2575 flags.go:64] FLAG: --register-node="true"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229154    2575 flags.go:64] FLAG: --register-with-taints=""
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229159    2575 flags.go:64] FLAG: --registry-burst="10"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229163    2575 flags.go:64] FLAG: --registry-qps="5"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229167    2575 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229172    2575 flags.go:64] FLAG: --reserved-memory=""
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229177    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229182    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229186    2575 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229203    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229208    2575 flags.go:64] FLAG: --runonce="false"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229213    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229217    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229222    2575 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229226    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229230    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229235    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229240    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229247    2575 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229251    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229256    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 18:30:08.235081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229260    2575 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229265    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229270    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229274    2575 flags.go:64] FLAG: --system-cgroups=""
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229278    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229287    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229291    2575 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229301    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229308    2575 flags.go:64] FLAG: --tls-min-version=""
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229315    2575 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229319    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229323    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229327    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229332    2575 flags.go:64] FLAG: --v="2"
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229338    2575 flags.go:64] FLAG: --version="false"
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229344    2575 flags.go:64] FLAG: --vmodule=""
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229350    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.229355    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229547    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229553    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229558    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229563    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229567    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 18:30:08.235712 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229570    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229574    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229580    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229585    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229588    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229593    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229599    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229603    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229608    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229612    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229615    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229619    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229623    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229627    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229631    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229635    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229638    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229642    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229646    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229650    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 18:30:08.236347 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229653    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229657    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229661    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229664    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229668    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229672    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229675    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229679    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229684    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229687    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229691    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229695    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229698    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229702    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229706    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229710    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229714    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229718    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229723    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229738    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 18:30:08.236856 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229743    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229747    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229750    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229754    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229759    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229763    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229766    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229770    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229773    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229777    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229781    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229785    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229789    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229793    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229796    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229799    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229804    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229807    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229811    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229815    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 18:30:08.237584 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229819    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229823    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229826    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229830    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229834    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229838    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229843    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229847    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229851    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229855    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229861    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229865    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229876    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229880    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229884    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229888    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229891    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229895    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229899    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 18:30:08.238104 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229903    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229907    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229911    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229915    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229919    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229923    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229926    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229930    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229934    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229938    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229941    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229945    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229948    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229952    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.229959    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 18:30:08.238600 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.230823    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 18:30:08.241313 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.241250    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 18:30:08.241313 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.241272    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 18:30:08.241313 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.241293    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 18:30:08.241313 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.241302    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241361    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241367    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241383    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241387    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241391    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241396    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241400    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241403    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241406    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241409    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241412    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241415    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241418    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241420    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241423    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241429    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241432    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241434    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241437    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 18:30:08.241479 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241440    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241443    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241445    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241448    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241451    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241453    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241456    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241458    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241461    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241463    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241466    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241469    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241473    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241476    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241479    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241482    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241485    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241488    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241491    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241494    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 18:30:08.241967 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241496    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241499    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241501    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241504    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241506    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241509    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241511    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241515    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241517    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241520    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241523    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241525    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241528    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241530    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241533    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241536    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241539    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241542    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241544    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241547    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 18:30:08.242486 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241549    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241552    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241554    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241557    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241559    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241562    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241565    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241567    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241570    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241573    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241575    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241578    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241580    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241583    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241585    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241588    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241590    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241593    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241596    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 18:30:08.242977 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241598    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241602    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241605    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241607    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241609    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241612    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241615    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241617    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241620    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241623    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241625    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241628    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241630    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241633    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241637    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241640    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241643    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241646    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241648    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241651    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 18:30:08.243460 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241654    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 18:30:08.243953 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.241659    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 18:30:08.243953 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241772    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 18:30:08.243953 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241776    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 18:30:08.243953 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241780    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 18:30:08.243953 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241782    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 18:30:08.243953 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241785    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 18:30:08.243953 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241790    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 18:30:08.243953 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241793    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 18:30:08.243953 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241796    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 18:30:08.243953 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241799    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 18:30:08.243953 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241801    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 18:30:08.243953 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241804    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 18:30:08.243953 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241806    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 18:30:08.243953 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241810    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241812    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241815    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241817    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241820    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241823    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241825    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241828    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241831    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241833    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241836    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241838    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241841    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241843    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241845    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241849    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241852    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241855    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241858    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 18:30:08.244326 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241861    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241864    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241866    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241869    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241871    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241874    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241876    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241879    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241881    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241884    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241886    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241889    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241891    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241893    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241897    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241899    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241902    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241904    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241908    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 18:30:08.244822 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241911    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241914    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241917    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241920    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241923    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241926    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241928    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241931    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241933    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241936    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241938    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241941    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241943    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241946    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241949    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241951    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241953    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241956    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241959    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241961    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 18:30:08.245370 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241964    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241966    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241969    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241971    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241974    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241976    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241979    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241982    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241986    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241988    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241991    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241993    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241996    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.241999    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242001    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242004    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242006    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242009    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242012    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242014    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 18:30:08.245847 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242017    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 18:30:08.246358 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242019    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 18:30:08.246358 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242021    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 18:30:08.246358 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242024    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 18:30:08.246358 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242026    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 18:30:08.246358 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242029    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 18:30:08.246358 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242032    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 18:30:08.246358 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242034    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 18:30:08.246358 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:08.242037    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 18:30:08.246358 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.242042    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 18:30:08.246358 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.242209    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 18:30:08.246358 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.244598    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 18:30:08.246358 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.246321    2575 server.go:1019] "Starting client certificate rotation"
Jul 02 18:30:08.246660 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.246441    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 18:30:08.246660 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.246482    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 18:30:08.250246 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.250231    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 18:30:08.251510 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.251495    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 18:30:08.260835 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.260817    2575 log.go:25] "Validated CRI v1 runtime API"
Jul 02 18:30:08.275915 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.275898    2575 log.go:25] "Validated CRI v1 image API"
Jul 02 18:30:08.276740 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.276719    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 18:30:08.279281 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.279261    2575 fs.go:135] Filesystem UUIDs: map[3bfd69e4-733e-4487-9c7a-307ae045c375:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e28d87e6-0faa-4ebb-943b-494c86324d41:/dev/nvme0n1p3]
Jul 02 18:30:08.279363 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.279279    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 18:30:08.282637 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.282617    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 18:30:08.284585 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.284470    2575 manager.go:216] Machine: {Timestamp:2026-07-02 18:30:08.283944534 +0000 UTC m=+0.353572052 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3072655 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2340245561864941dab018b2828956 SystemUUID:ec234024-5561-8649-41da-b018b2828956 BootID:851a49ec-65de-47b3-a0b4-9c96394ece60 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b9:5b:5e:d5:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b9:5b:5e:d5:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:8b:22:ad:eb:b8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 18:30:08.284585 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.284582    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 18:30:08.284697 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.284650    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 18:30:08.289504 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.289477    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 18:30:08.289639 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.289505    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-68.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 18:30:08.290362 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.290352    2575 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 18:30:08.290395 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.290365    2575 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 18:30:08.290395 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.290373    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 18:30:08.291223 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.291212    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 18:30:08.292908 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.292897    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 18:30:08.296359 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.296348    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 18:30:08.296482 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.296472    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 18:30:08.296511 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.296505    2575 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 18:30:08.296538 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.296520    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 18:30:08.296575 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.296544    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 18:30:08.296575 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.296557    2575 kubelet.go:398] "Adding apiserver pod source"
Jul 02 18:30:08.296631 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.296577    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 18:30:08.296994 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.296983    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 18:30:08.297078 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.297001    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 18:30:08.300030 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.300013    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 18:30:08.302568 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.302552    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 18:30:08.302664 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.302591    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 18:30:08.304672 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.304658    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 18:30:08.304751 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.304683    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 18:30:08.304751 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.304693    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 18:30:08.304751 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.304701    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 18:30:08.304751 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.304712    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 18:30:08.304751 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.304720    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 18:30:08.304751 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.304731    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 18:30:08.304751 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.304740    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 18:30:08.304751 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.304751    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 18:30:08.304992 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.304759    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 18:30:08.304992 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.304772    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 18:30:08.304992 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.304781    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 18:30:08.305765 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.305754    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 18:30:08.305821 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.305768    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 18:30:08.309040 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.309024    2575 server.go:1268] "Started kubelet"
Jul 02 18:30:08.309151 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.309102    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 18:30:08.309225 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.309179    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 18:30:08.309277 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.309168    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 18:30:08.310027 ip-10-0-140-68 systemd[1]: Started Kubernetes Kubelet.
Jul 02 18:30:08.311433 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.311416    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 18:30:08.312179 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.312162    2575 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 18:30:08.313466 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.313428    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 18:30:08.313560 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.313464    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 18:30:08.314396 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.314376    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-68.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 18:30:08.315404 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.315375    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 18:30:08.320036 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.318925    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-68.ec2.internal.18be8c85b0bce30a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-68.ec2.internal,UID:ip-10-0-140-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-68.ec2.internal,},FirstTimestamp:2026-07-02 18:30:08.309003018 +0000 UTC m=+0.378630537,LastTimestamp:2026-07-02 18:30:08.309003018 +0000 UTC m=+0.378630537,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-68.ec2.internal,}"
Jul 02 18:30:08.320994 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.320974    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 18:30:08.320994 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.320991    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 18:30:08.321172 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.321134    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Jul 02 18:30:08.322149 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.322101    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 18:30:08.322149 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.322133    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 18:30:08.322287 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.322255    2575 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 18:30:08.322287 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.322266    2575 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 18:30:08.322387 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.322342    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 18:30:08.322387 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.322357    2575 factory.go:55] Registering systemd factory
Jul 02 18:30:08.322387 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.322367    2575 factory.go:223] Registration of the systemd container factory successfully
Jul 02 18:30:08.322537 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.322525    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 18:30:08.322608 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.322592    2575 factory.go:153] Registering CRI-O factory
Jul 02 18:30:08.322608 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.322609    2575 factory.go:223] Registration of the crio container factory successfully
Jul 02 18:30:08.322737 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.322633    2575 factory.go:103] Registering Raw factory
Jul 02 18:30:08.322737 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.322646    2575 manager.go:1196] Started watching for new ooms in manager
Jul 02 18:30:08.323313 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.323297    2575 manager.go:349] Starting recovery of all containers
Jul 02 18:30:08.328977 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.328947    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 18:30:08.329126 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.329101    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 18:30:08.335902 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.335739    2575 manager.go:354] Recovery completed
Jul 02 18:30:08.340082 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.340070    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 18:30:08.341992 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.341968    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 18:30:08.342064 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.342009    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 18:30:08.342064 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.342021    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientPID"
Jul 02 18:30:08.342569 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.342557    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 18:30:08.342569 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.342564    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 18:30:08.342569 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.342568    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 18:30:08.342657 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.342583    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 18:30:08.343790 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.343765    2575 policy_none.go:49] "None policy: Start"
Jul 02 18:30:08.343790 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.343784    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 18:30:08.343923 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.343797    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 18:30:08.343923 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.343803    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 18:30:08.343923 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.343826    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 18:30:08.343923 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.343840    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientPID"
Jul 02 18:30:08.344027 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.343915    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-68.ec2.internal.18be8c85b2b44ced  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-68.ec2.internal,UID:ip-10-0-140-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-68.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-68.ec2.internal,},FirstTimestamp:2026-07-02 18:30:08.341994733 +0000 UTC m=+0.411622256,LastTimestamp:2026-07-02 18:30:08.341994733 +0000 UTC m=+0.411622256,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-68.ec2.internal,}"
Jul 02 18:30:08.344789 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.344777    2575 policy_none.go:47] "Start"
Jul 02 18:30:08.359605 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.359547    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-68.ec2.internal.18be8c85b2b49bc6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-68.ec2.internal,UID:ip-10-0-140-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-68.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-68.ec2.internal,},FirstTimestamp:2026-07-02 18:30:08.342014918 +0000 UTC m=+0.411642437,LastTimestamp:2026-07-02 18:30:08.342014918 +0000 UTC m=+0.411642437,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-68.ec2.internal,}"
Jul 02 18:30:08.370767 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.370691    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-68.ec2.internal.18be8c85b2b4c1de  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-68.ec2.internal,UID:ip-10-0-140-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-68.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-68.ec2.internal,},FirstTimestamp:2026-07-02 18:30:08.34202467 +0000 UTC m=+0.411652189,LastTimestamp:2026-07-02 18:30:08.34202467 +0000 UTC m=+0.411652189,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-68.ec2.internal,}"
Jul 02 18:30:08.371297 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.371167    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qppz8"
Jul 02 18:30:08.377556 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.377535    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qppz8"
Jul 02 18:30:08.384830 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.384815    2575 manager.go:337] "Starting Device Plugin manager"
Jul 02 18:30:08.399530 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.384852    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 18:30:08.399530 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.384866    2575 server.go:87] "Starting device plugin registration server"
Jul 02 18:30:08.399530 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.385111    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 18:30:08.399530 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.385126    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 18:30:08.399530 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.385237    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 18:30:08.399530 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.385310    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 18:30:08.399530 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.385319    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 18:30:08.399530 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.385805    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 18:30:08.399530 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.385860    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-68.ec2.internal\" not found"
Jul 02 18:30:08.481409 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.481323    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 18:30:08.482652 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.482632    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 18:30:08.482737 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.482671    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 18:30:08.482737 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.482693    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 18:30:08.482737 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.482725    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 18:30:08.484836 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.484808    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 18:30:08.485330 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.485315    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 18:30:08.486151 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.486136    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 18:30:08.486253 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.486165    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 18:30:08.486253 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.486176    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientPID"
Jul 02 18:30:08.486253 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.486215    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.501460 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.501439    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.501552 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.501466    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-68.ec2.internal\": node \"ip-10-0-140-68.ec2.internal\" not found"
Jul 02 18:30:08.526494 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.526463    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Jul 02 18:30:08.583479 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.583442    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"]
Jul 02 18:30:08.583605 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.583537    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 18:30:08.584560 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.584541    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 18:30:08.584658 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.584583    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 18:30:08.584658 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.584594    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientPID"
Jul 02 18:30:08.585840 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.585827    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 18:30:08.585947 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.585934    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.585982 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.585963    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 18:30:08.586526 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.586508    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 18:30:08.586608 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.586537    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 18:30:08.586608 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.586547    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientPID"
Jul 02 18:30:08.586608 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.586511    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 18:30:08.586702 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.586607    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 18:30:08.586702 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.586624    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientPID"
Jul 02 18:30:08.589645 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.589631    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.589695 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.589656    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 18:30:08.590324 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.590309    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 18:30:08.590378 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.590335    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 18:30:08.590378 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.590345    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientPID"
Jul 02 18:30:08.617301 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.617273    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-68.ec2.internal\" not found" node="ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.621720 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.621701    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-68.ec2.internal\" not found" node="ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.623837 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.623816    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06004644724e4a334662d055fee9ec18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal\" (UID: \"06004644724e4a334662d055fee9ec18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.623922 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.623842    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06004644724e4a334662d055fee9ec18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal\" (UID: \"06004644724e4a334662d055fee9ec18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.623922 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.623860    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/000fc49438e162666c8381fedce28b6d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-68.ec2.internal\" (UID: \"000fc49438e162666c8381fedce28b6d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.626994 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.626979    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Jul 02 18:30:08.724927 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.724887    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06004644724e4a334662d055fee9ec18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal\" (UID: \"06004644724e4a334662d055fee9ec18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.724927 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.724928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06004644724e4a334662d055fee9ec18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal\" (UID: \"06004644724e4a334662d055fee9ec18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.725092 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.724945    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/000fc49438e162666c8381fedce28b6d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-68.ec2.internal\" (UID: \"000fc49438e162666c8381fedce28b6d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.725092 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.724977    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/000fc49438e162666c8381fedce28b6d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-68.ec2.internal\" (UID: \"000fc49438e162666c8381fedce28b6d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.725092 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.725018    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06004644724e4a334662d055fee9ec18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal\" (UID: \"06004644724e4a334662d055fee9ec18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.725092 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.725018    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06004644724e4a334662d055fee9ec18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal\" (UID: \"06004644724e4a334662d055fee9ec18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.727105 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.727086    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Jul 02 18:30:08.827317 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.827288    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Jul 02 18:30:08.924287 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.924249    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.925456 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:08.925437    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:08.928703 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:08.927813    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Jul 02 18:30:09.029020 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:09.028966    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Jul 02 18:30:09.129999 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:09.129918    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Jul 02 18:30:09.184727 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.184700    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 18:30:09.222145 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.222123    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:09.235169 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.235144    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 18:30:09.237160 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.237145    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:09.244993 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.244972    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 18:30:09.245158 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.245134    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 18:30:09.245233 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.245134    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 18:30:09.245233 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:09.245170    2575 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://ae00b0a575fcf49a3911c11bae80efbc-4981c18b028d9b59.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.140.68:55900->184.73.215.78:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"
Jul 02 18:30:09.300403 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.300375    2575 apiserver.go:52] "Watching apiserver"
Jul 02 18:30:09.319574 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.319550    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 18:30:09.321078 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.321059    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 18:30:09.322642 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.322615    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-b7jnx","openshift-network-operator/iptables-alerter-m8sht","openshift-ovn-kubernetes/ovnkube-node-vkdd4","kube-system/global-pull-secret-syncer-5j4fb","kube-system/konnectivity-agent-7wdxk","kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc","openshift-dns/node-resolver-6mtkp","openshift-image-registry/node-ca-6cwjc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal","openshift-multus/multus-additional-cni-plugins-6f6sb","openshift-cluster-node-tuning-operator/tuned-hhvg5","openshift-multus/multus-rglzr","openshift-multus/network-metrics-daemon-m2h4c"]
Jul 02 18:30:09.323989 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.323972    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:09.324086 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:09.324054    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:09.325003 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.324988    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m8sht"
Jul 02 18:30:09.326236 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.326215    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.327327 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.327308    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:09.327410 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:09.327377    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:09.327798 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.327782    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 18:30:09.327867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.327821    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 18:30:09.328621 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.328600    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 18:30:09.328725 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.328619    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kzbcl\""
Jul 02 18:30:09.328725 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.328666    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7wdxk"
Jul 02 18:30:09.328869 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.328853    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 18:30:09.328955 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.328939    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 18:30:09.329931 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.329914    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.331038 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.331021    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6mtkp"
Jul 02 18:30:09.331191 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.331174    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-td6xm\""
Jul 02 18:30:09.331334 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.331317    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 18:30:09.331444 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.331428    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 18:30:09.331510 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.331442    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 18:30:09.331510 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.331488    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 18:30:09.332165 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.332148    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 18:30:09.332238 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.332226    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 18:30:09.332494 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.332378    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-68526\""
Jul 02 18:30:09.332494 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.332437    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6cwjc"
Jul 02 18:30:09.333221 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.332920    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 18:30:09.333221 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.332995    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 18:30:09.333221 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.333050    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 18:30:09.333221 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.333117    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jb8dr\""
Jul 02 18:30:09.333727 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.333715    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 18:30:09.333952 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.333936    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 18:30:09.334009 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.333995    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.334595 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.334580    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j4qgk\""
Jul 02 18:30:09.335034 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.335020    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 18:30:09.335494 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.335362    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.336223 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.335802    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-26cnz\""
Jul 02 18:30:09.336223 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.335893    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 18:30:09.337172 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.336907    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 18:30:09.337172 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.337009    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 18:30:09.337172 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.337078    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 18:30:09.337921 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.337309    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2vtln\""
Jul 02 18:30:09.337921 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.337391    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 18:30:09.337921 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.337471    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 18:30:09.337921 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.337595    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 18:30:09.337921 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.337671    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 18:30:09.337921 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.337904    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.339531 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.339206    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 18:30:09.339531 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.339412    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 18:30:09.339531 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.339431    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-644jm\""
Jul 02 18:30:09.340164 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.340144    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:09.340260 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:09.340216    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:09.342574 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.342559    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dx2f4\""
Jul 02 18:30:09.342633 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.342606    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 18:30:09.356612 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.356581    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4htrr"
Jul 02 18:30:09.368306 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.368277    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4htrr"
Jul 02 18:30:09.379988 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.379925    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 18:25:08 +0000 UTC" deadline="2028-01-03 04:04:06.390607299 +0000 UTC"
Jul 02 18:30:09.379988 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.379983    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13185h33m57.010628833s"
Jul 02 18:30:09.424546 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.424521    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 18:30:09.430011 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.429990    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-var-lib-cni-multus\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.430112 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430021    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-run-multus-certs\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.430112 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430045    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9qfp\" (UniqueName: \"kubernetes.io/projected/d5a96438-0081-4f62-aa03-cd7cf1bb0f2c-kube-api-access-w9qfp\") pod \"node-resolver-6mtkp\" (UID: \"d5a96438-0081-4f62-aa03-cd7cf1bb0f2c\") " pod="openshift-dns/node-resolver-6mtkp"
Jul 02 18:30:09.430112 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430068    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxndc\" (UniqueName: \"kubernetes.io/projected/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-kube-api-access-zxndc\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.430112 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430100    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-var-lib-kubelet\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.430351 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430139    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-slash\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.430351 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430162    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.430351 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430183    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjsh9\" (UniqueName: \"kubernetes.io/projected/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-kube-api-access-pjsh9\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.430351 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430209    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3682ee4-82ef-4457-a048-4de940c25ac0-kubelet-config\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:09.430351 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430232    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwr9l\" (UniqueName: \"kubernetes.io/projected/02cf5868-e870-47eb-870f-c87dcabdae5a-kube-api-access-dwr9l\") pod \"node-ca-6cwjc\" (UID: \"02cf5868-e870-47eb-870f-c87dcabdae5a\") " pod="openshift-image-registry/node-ca-6cwjc"
Jul 02 18:30:09.430351 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430251    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-hostroot\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.430351 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-multus-conf-dir\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.430351 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430303    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-system-cni-dir\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.430351 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-modprobe-d\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430355    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-run\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430380    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e84ffa74-4554-441b-8e99-be4646050843-cni-binary-copy\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430404    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-var-lib-cni-bin\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430426    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72fc1e32-b32e-415f-9101-62498e24ea7b-agent-certs\") pod \"konnectivity-agent-7wdxk\" (UID: \"72fc1e32-b32e-415f-9101-62498e24ea7b\") " pod="kube-system/konnectivity-agent-7wdxk"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430451    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-sys-fs\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430473    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-sys\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430499    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-etc-kubernetes\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430519    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430541    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-run-netns\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430557    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430576    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-cnibin\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-os-release\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430627    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430658    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-kubernetes\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.430698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430687    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-sysctl-d\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430728    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-run-netns\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430746    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crrbm\" (UniqueName: \"kubernetes.io/projected/e84ffa74-4554-441b-8e99-be4646050843-kube-api-access-crrbm\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430763    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430791    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-node-log\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430835    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a96438-0081-4f62-aa03-cd7cf1bb0f2c-tmp-dir\") pod \"node-resolver-6mtkp\" (UID: \"d5a96438-0081-4f62-aa03-cd7cf1bb0f2c\") " pod="openshift-dns/node-resolver-6mtkp"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430874    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-tuned\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430930    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-ovn-node-metrics-cert\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430956    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-ovnkube-script-lib\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.430985    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsjkn\" (UniqueName: \"kubernetes.io/projected/f1197266-26d4-4a4d-9652-0db22827bf3b-kube-api-access-tsjkn\") pod \"iptables-alerter-m8sht\" (UID: \"f1197266-26d4-4a4d-9652-0db22827bf3b\") " pod="openshift-network-operator/iptables-alerter-m8sht"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431006    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-systemd\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431023    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-etc-openvswitch\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431038    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-cni-netd\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431063    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72fc1e32-b32e-415f-9101-62498e24ea7b-konnectivity-ca\") pod \"konnectivity-agent-7wdxk\" (UID: \"72fc1e32-b32e-415f-9101-62498e24ea7b\") " pod="kube-system/konnectivity-agent-7wdxk"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431078    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-run-openvswitch\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.431177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431107    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3682ee4-82ef-4457-a048-4de940c25ac0-dbus\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431139    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02cf5868-e870-47eb-870f-c87dcabdae5a-serviceca\") pod \"node-ca-6cwjc\" (UID: \"02cf5868-e870-47eb-870f-c87dcabdae5a\") " pod="openshift-image-registry/node-ca-6cwjc"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431168    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-os-release\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431214    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs\") pod \"network-metrics-daemon-m2h4c\" (UID: \"780da6ef-b5e6-43c9-b539-185cc5434c63\") " pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431240    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1197266-26d4-4a4d-9652-0db22827bf3b-host-slash\") pod \"iptables-alerter-m8sht\" (UID: \"f1197266-26d4-4a4d-9652-0db22827bf3b\") " pod="openshift-network-operator/iptables-alerter-m8sht"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431265    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431304    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-kubelet\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431340    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-systemd-units\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431368    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431399    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02cf5868-e870-47eb-870f-c87dcabdae5a-host\") pod \"node-ca-6cwjc\" (UID: \"02cf5868-e870-47eb-870f-c87dcabdae5a\") " pod="openshift-image-registry/node-ca-6cwjc"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431425    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxwj6\" (UniqueName: \"kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6\") pod \"network-check-target-b7jnx\" (UID: \"7340eb6c-cda8-42dc-9abe-2b16be84f537\") " pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431450    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d5a96438-0081-4f62-aa03-cd7cf1bb0f2c-hosts-file\") pod \"node-resolver-6mtkp\" (UID: \"d5a96438-0081-4f62-aa03-cd7cf1bb0f2c\") " pod="openshift-dns/node-resolver-6mtkp"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431480    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-cnibin\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431500    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-cni-bin\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431524    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-run-k8s-cni-cncf-io\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431542    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ktzw\" (UniqueName: \"kubernetes.io/projected/780da6ef-b5e6-43c9-b539-185cc5434c63-kube-api-access-9ktzw\") pod \"network-metrics-daemon-m2h4c\" (UID: \"780da6ef-b5e6-43c9-b539-185cc5434c63\") " pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:09.431904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431564    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1197266-26d4-4a4d-9652-0db22827bf3b-iptables-alerter-script\") pod \"iptables-alerter-m8sht\" (UID: \"f1197266-26d4-4a4d-9652-0db22827bf3b\") " pod="openshift-network-operator/iptables-alerter-m8sht"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431582    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-device-dir\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431623    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvjft\" (UniqueName: \"kubernetes.io/projected/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-kube-api-access-dvjft\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431652    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-lib-modules\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-host\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431705    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-system-cni-dir\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431732    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-var-lib-kubelet\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431759    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-run-systemd\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431785    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e84ffa74-4554-441b-8e99-be4646050843-multus-daemon-config\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431810    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-registration-dir\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431835    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-tmp\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431859    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-var-lib-openvswitch\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431884    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-run-ovn\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431929    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-env-overrides\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431953    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-multus-socket-dir-parent\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.431989    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.432490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.432005    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jdnz\" (UniqueName: \"kubernetes.io/projected/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-kube-api-access-2jdnz\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.433015 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.432020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-multus-cni-dir\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.433015 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.432056    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-socket-dir\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.433015 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.432070    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-sysconfig\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.433015 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.432107    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-sysctl-conf\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.433015 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.432136    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-log-socket\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.433015 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.432152    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-ovnkube-config\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.521847 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.521820    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 18:30:09.522263 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:09.522214    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000fc49438e162666c8381fedce28b6d.slice/crio-5b77f8a8b65cdfd62f33ecce3c13dd4694000e4995b607bd083dd1ac7891fb01 WatchSource:0}: Error finding container 5b77f8a8b65cdfd62f33ecce3c13dd4694000e4995b607bd083dd1ac7891fb01: Status 404 returned error can't find the container with id 5b77f8a8b65cdfd62f33ecce3c13dd4694000e4995b607bd083dd1ac7891fb01
Jul 02 18:30:09.522618 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:09.522590    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06004644724e4a334662d055fee9ec18.slice/crio-3cea103836085c1e3e42b24f0d4db16479cbfcfabdff6cbf5ca8b6809e39ca65 WatchSource:0}: Error finding container 3cea103836085c1e3e42b24f0d4db16479cbfcfabdff6cbf5ca8b6809e39ca65: Status 404 returned error can't find the container with id 3cea103836085c1e3e42b24f0d4db16479cbfcfabdff6cbf5ca8b6809e39ca65
Jul 02 18:30:09.527822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.527798    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 18:30:09.533221 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533197    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-socket-dir\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.533327 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533239    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-sysconfig\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.533327 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533265    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-sysctl-conf\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.533327 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533308    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-log-socket\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.533489 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533333    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-ovnkube-config\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.533489 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533352    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-sysconfig\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.533489 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-var-lib-cni-multus\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.533489 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533403    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-socket-dir\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.533489 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533411    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-run-multus-certs\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.533489 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533425    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-log-socket\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.533489 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533433    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9qfp\" (UniqueName: \"kubernetes.io/projected/d5a96438-0081-4f62-aa03-cd7cf1bb0f2c-kube-api-access-w9qfp\") pod \"node-resolver-6mtkp\" (UID: \"d5a96438-0081-4f62-aa03-cd7cf1bb0f2c\") " pod="openshift-dns/node-resolver-6mtkp"
Jul 02 18:30:09.533489 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533460    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-run-multus-certs\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.533489 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxndc\" (UniqueName: \"kubernetes.io/projected/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-kube-api-access-zxndc\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533494    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-var-lib-cni-multus\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533519    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-sysctl-conf\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533545    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-var-lib-kubelet\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-slash\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533592    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-var-lib-kubelet\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533606    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533632    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-slash\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533637    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjsh9\" (UniqueName: \"kubernetes.io/projected/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-kube-api-access-pjsh9\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3682ee4-82ef-4457-a048-4de940c25ac0-kubelet-config\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533674    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533691    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwr9l\" (UniqueName: \"kubernetes.io/projected/02cf5868-e870-47eb-870f-c87dcabdae5a-kube-api-access-dwr9l\") pod \"node-ca-6cwjc\" (UID: \"02cf5868-e870-47eb-870f-c87dcabdae5a\") " pod="openshift-image-registry/node-ca-6cwjc"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-hostroot\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533738    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3682ee4-82ef-4457-a048-4de940c25ac0-kubelet-config\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533759    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-hostroot\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-multus-conf-dir\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533867    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-system-cni-dir\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.533940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533893    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-modprobe-d\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533922    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-run\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533933    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-system-cni-dir\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533949    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e84ffa74-4554-441b-8e99-be4646050843-cni-binary-copy\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533937    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-ovnkube-config\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533974    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-var-lib-cni-bin\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533983    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-run\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533995    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-modprobe-d\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533923    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-multus-conf-dir\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.533999    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72fc1e32-b32e-415f-9101-62498e24ea7b-agent-certs\") pod \"konnectivity-agent-7wdxk\" (UID: \"72fc1e32-b32e-415f-9101-62498e24ea7b\") " pod="kube-system/konnectivity-agent-7wdxk"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534042    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-var-lib-cni-bin\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534069    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-sys-fs\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534096    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-sys\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534122    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-etc-kubernetes\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534149    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-sys-fs\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534169    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534175    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-sys\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534188    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-etc-kubernetes\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.534838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534194    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-run-netns\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534249    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-run-netns\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534280    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534283    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-cnibin\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534330    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-os-release\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534348    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-cnibin\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534358    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534390    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-kubernetes\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534413    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-os-release\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534418    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-sysctl-d\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534453    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-kubernetes\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534454    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-run-netns\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534494    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crrbm\" (UniqueName: \"kubernetes.io/projected/e84ffa74-4554-441b-8e99-be4646050843-kube-api-access-crrbm\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534524    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534528    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-sysctl-d\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.535644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534551    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-run-netns\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534553    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-node-log\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534593    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a96438-0081-4f62-aa03-cd7cf1bb0f2c-tmp-dir\") pod \"node-resolver-6mtkp\" (UID: \"d5a96438-0081-4f62-aa03-cd7cf1bb0f2c\") " pod="openshift-dns/node-resolver-6mtkp"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534595    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-node-log\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534594    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e84ffa74-4554-441b-8e99-be4646050843-cni-binary-copy\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534692    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-tuned\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534936    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a96438-0081-4f62-aa03-cd7cf1bb0f2c-tmp-dir\") pod \"node-resolver-6mtkp\" (UID: \"d5a96438-0081-4f62-aa03-cd7cf1bb0f2c\") " pod="openshift-dns/node-resolver-6mtkp"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534968    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.534983    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535005    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-ovn-node-metrics-cert\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-ovnkube-script-lib\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535041    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsjkn\" (UniqueName: \"kubernetes.io/projected/f1197266-26d4-4a4d-9652-0db22827bf3b-kube-api-access-tsjkn\") pod \"iptables-alerter-m8sht\" (UID: \"f1197266-26d4-4a4d-9652-0db22827bf3b\") " pod="openshift-network-operator/iptables-alerter-m8sht"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535065    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-systemd\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535101    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-etc-openvswitch\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-cni-netd\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535155    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72fc1e32-b32e-415f-9101-62498e24ea7b-konnectivity-ca\") pod \"konnectivity-agent-7wdxk\" (UID: \"72fc1e32-b32e-415f-9101-62498e24ea7b\") " pod="kube-system/konnectivity-agent-7wdxk"
Jul 02 18:30:09.536513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-run-openvswitch\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535202    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3682ee4-82ef-4457-a048-4de940c25ac0-dbus\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535251    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02cf5868-e870-47eb-870f-c87dcabdae5a-serviceca\") pod \"node-ca-6cwjc\" (UID: \"02cf5868-e870-47eb-870f-c87dcabdae5a\") " pod="openshift-image-registry/node-ca-6cwjc"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535264    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-os-release\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535324    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs\") pod \"network-metrics-daemon-m2h4c\" (UID: \"780da6ef-b5e6-43c9-b539-185cc5434c63\") " pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535333    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-os-release\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535355    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-systemd\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:09.535408    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:09.535463    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs podName:780da6ef-b5e6-43c9-b539-185cc5434c63 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:10.035442554 +0000 UTC m=+2.105070088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs") pod "network-metrics-daemon-m2h4c" (UID: "780da6ef-b5e6-43c9-b539-185cc5434c63") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535521    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3682ee4-82ef-4457-a048-4de940c25ac0-dbus\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535569    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-etc-openvswitch\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535611    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-cni-netd\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535627    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-ovnkube-script-lib\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535643    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1197266-26d4-4a4d-9652-0db22827bf3b-host-slash\") pod \"iptables-alerter-m8sht\" (UID: \"f1197266-26d4-4a4d-9652-0db22827bf3b\") " pod="openshift-network-operator/iptables-alerter-m8sht"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535676    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1197266-26d4-4a4d-9652-0db22827bf3b-host-slash\") pod \"iptables-alerter-m8sht\" (UID: \"f1197266-26d4-4a4d-9652-0db22827bf3b\") " pod="openshift-network-operator/iptables-alerter-m8sht"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535679    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.537404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535715    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-run-openvswitch\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535735    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-kubelet\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535772    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-systemd-units\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535789    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02cf5868-e870-47eb-870f-c87dcabdae5a-serviceca\") pod \"node-ca-6cwjc\" (UID: \"02cf5868-e870-47eb-870f-c87dcabdae5a\") " pod="openshift-image-registry/node-ca-6cwjc"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535798    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535823    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02cf5868-e870-47eb-870f-c87dcabdae5a-host\") pod \"node-ca-6cwjc\" (UID: \"02cf5868-e870-47eb-870f-c87dcabdae5a\") " pod="openshift-image-registry/node-ca-6cwjc"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535851    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwj6\" (UniqueName: \"kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6\") pod \"network-check-target-b7jnx\" (UID: \"7340eb6c-cda8-42dc-9abe-2b16be84f537\") " pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535878    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d5a96438-0081-4f62-aa03-cd7cf1bb0f2c-hosts-file\") pod \"node-resolver-6mtkp\" (UID: \"d5a96438-0081-4f62-aa03-cd7cf1bb0f2c\") " pod="openshift-dns/node-resolver-6mtkp"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:09.535883    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535883    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02cf5868-e870-47eb-870f-c87dcabdae5a-host\") pod \"node-ca-6cwjc\" (UID: \"02cf5868-e870-47eb-870f-c87dcabdae5a\") " pod="openshift-image-registry/node-ca-6cwjc"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535852    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-kubelet\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535904    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-cnibin\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:09.535940    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret podName:d3682ee4-82ef-4457-a048-4de940c25ac0 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:10.035924514 +0000 UTC m=+2.105552040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret") pod "global-pull-secret-syncer-5j4fb" (UID: "d3682ee4-82ef-4457-a048-4de940c25ac0") : object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535959    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-cni-bin\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535980    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72fc1e32-b32e-415f-9101-62498e24ea7b-konnectivity-ca\") pod \"konnectivity-agent-7wdxk\" (UID: \"72fc1e32-b32e-415f-9101-62498e24ea7b\") " pod="kube-system/konnectivity-agent-7wdxk"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535993    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-run-k8s-cni-cncf-io\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.538217 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.535996    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d5a96438-0081-4f62-aa03-cd7cf1bb0f2c-hosts-file\") pod \"node-resolver-6mtkp\" (UID: \"d5a96438-0081-4f62-aa03-cd7cf1bb0f2c\") " pod="openshift-dns/node-resolver-6mtkp"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536041    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ktzw\" (UniqueName: \"kubernetes.io/projected/780da6ef-b5e6-43c9-b539-185cc5434c63-kube-api-access-9ktzw\") pod \"network-metrics-daemon-m2h4c\" (UID: \"780da6ef-b5e6-43c9-b539-185cc5434c63\") " pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536046    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-run-k8s-cni-cncf-io\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536052    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-systemd-units\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536085    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-host-cni-bin\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536092    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-cnibin\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536118    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1197266-26d4-4a4d-9652-0db22827bf3b-iptables-alerter-script\") pod \"iptables-alerter-m8sht\" (UID: \"f1197266-26d4-4a4d-9652-0db22827bf3b\") " pod="openshift-network-operator/iptables-alerter-m8sht"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536247    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-device-dir\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536312    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvjft\" (UniqueName: \"kubernetes.io/projected/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-kube-api-access-dvjft\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536339    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-lib-modules\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536362    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-device-dir\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536363    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-host\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-system-cni-dir\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536600    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-host\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536465    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-var-lib-kubelet\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536661    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-run-systemd\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536690    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e84ffa74-4554-441b-8e99-be4646050843-multus-daemon-config\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.538817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536701    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-lib-modules\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-registration-dir\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536741    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-tmp\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536758    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1197266-26d4-4a4d-9652-0db22827bf3b-iptables-alerter-script\") pod \"iptables-alerter-m8sht\" (UID: \"f1197266-26d4-4a4d-9652-0db22827bf3b\") " pod="openshift-network-operator/iptables-alerter-m8sht"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536780    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-host-var-lib-kubelet\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-var-lib-openvswitch\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536789    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-system-cni-dir\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-run-ovn\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536824    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-var-lib-openvswitch\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536852    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-registration-dir\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-env-overrides\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536866    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-run-systemd\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536896    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-multus-socket-dir-parent\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536900    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-run-ovn\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.536956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jdnz\" (UniqueName: \"kubernetes.io/projected/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-kube-api-access-2jdnz\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.537004    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-multus-cni-dir\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.539350 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.537088    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-multus-cni-dir\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.539867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.537141    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e84ffa74-4554-441b-8e99-be4646050843-multus-socket-dir-parent\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.539867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.537298    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.539867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.537326    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-env-overrides\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.539867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.537330    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e84ffa74-4554-441b-8e99-be4646050843-multus-daemon-config\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.539867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.538334    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-etc-tuned\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.539867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.538539    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-ovn-node-metrics-cert\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.539867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.538638    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72fc1e32-b32e-415f-9101-62498e24ea7b-agent-certs\") pod \"konnectivity-agent-7wdxk\" (UID: \"72fc1e32-b32e-415f-9101-62498e24ea7b\") " pod="kube-system/konnectivity-agent-7wdxk"
Jul 02 18:30:09.539867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.539019    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-tmp\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.547412 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:09.547278    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 18:30:09.547412 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:09.547303    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 18:30:09.547412 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:09.547317    2575 projected.go:196] Error preparing data for projected volume kube-api-access-sxwj6 for pod openshift-network-diagnostics/network-check-target-b7jnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:09.547412 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:09.547379    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6 podName:7340eb6c-cda8-42dc-9abe-2b16be84f537 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:10.047362972 +0000 UTC m=+2.116990501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sxwj6" (UniqueName: "kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6") pod "network-check-target-b7jnx" (UID: "7340eb6c-cda8-42dc-9abe-2b16be84f537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:09.547797 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.547604    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9qfp\" (UniqueName: \"kubernetes.io/projected/d5a96438-0081-4f62-aa03-cd7cf1bb0f2c-kube-api-access-w9qfp\") pod \"node-resolver-6mtkp\" (UID: \"d5a96438-0081-4f62-aa03-cd7cf1bb0f2c\") " pod="openshift-dns/node-resolver-6mtkp"
Jul 02 18:30:09.548793 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.548766    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxndc\" (UniqueName: \"kubernetes.io/projected/98e2289a-1242-48fa-9b3e-a89b46c2c9d8-kube-api-access-zxndc\") pod \"multus-additional-cni-plugins-6f6sb\" (UID: \"98e2289a-1242-48fa-9b3e-a89b46c2c9d8\") " pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.549725 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.549697    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jdnz\" (UniqueName: \"kubernetes.io/projected/c01f69e9-cff9-4c5a-97d0-68a53d05b20b-kube-api-access-2jdnz\") pod \"tuned-hhvg5\" (UID: \"c01f69e9-cff9-4c5a-97d0-68a53d05b20b\") " pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.549838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.549815    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvjft\" (UniqueName: \"kubernetes.io/projected/a0c1fe04-027c-473f-ad0a-0cabf800d9e3-kube-api-access-dvjft\") pod \"aws-ebs-csi-driver-node-48csc\" (UID: \"a0c1fe04-027c-473f-ad0a-0cabf800d9e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.549976 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.549959    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ktzw\" (UniqueName: \"kubernetes.io/projected/780da6ef-b5e6-43c9-b539-185cc5434c63-kube-api-access-9ktzw\") pod \"network-metrics-daemon-m2h4c\" (UID: \"780da6ef-b5e6-43c9-b539-185cc5434c63\") " pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:09.550161 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.550140    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crrbm\" (UniqueName: \"kubernetes.io/projected/e84ffa74-4554-441b-8e99-be4646050843-kube-api-access-crrbm\") pod \"multus-rglzr\" (UID: \"e84ffa74-4554-441b-8e99-be4646050843\") " pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.550550 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.550529    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwr9l\" (UniqueName: \"kubernetes.io/projected/02cf5868-e870-47eb-870f-c87dcabdae5a-kube-api-access-dwr9l\") pod \"node-ca-6cwjc\" (UID: \"02cf5868-e870-47eb-870f-c87dcabdae5a\") " pod="openshift-image-registry/node-ca-6cwjc"
Jul 02 18:30:09.550803 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.550788    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjsh9\" (UniqueName: \"kubernetes.io/projected/0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b-kube-api-access-pjsh9\") pod \"ovnkube-node-vkdd4\" (UID: \"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.551085 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.551068    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsjkn\" (UniqueName: \"kubernetes.io/projected/f1197266-26d4-4a4d-9652-0db22827bf3b-kube-api-access-tsjkn\") pod \"iptables-alerter-m8sht\" (UID: \"f1197266-26d4-4a4d-9652-0db22827bf3b\") " pod="openshift-network-operator/iptables-alerter-m8sht"
Jul 02 18:30:09.655517 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.655432    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m8sht"
Jul 02 18:30:09.665442 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:09.665419    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1197266_26d4_4a4d_9652_0db22827bf3b.slice/crio-df47527ecc6de901166c1182f08b12cbb7119ed27136bd353731389c7e93f78e WatchSource:0}: Error finding container df47527ecc6de901166c1182f08b12cbb7119ed27136bd353731389c7e93f78e: Status 404 returned error can't find the container with id df47527ecc6de901166c1182f08b12cbb7119ed27136bd353731389c7e93f78e
Jul 02 18:30:09.673743 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.673718    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:09.681475 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:09.681450    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4b9b59_08ac_4d0b_a98c_48b6a4881c2b.slice/crio-685e8f13695bf51f37b393d593c7108dd7a1a2032cd905e1cd8b421ef46c3807 WatchSource:0}: Error finding container 685e8f13695bf51f37b393d593c7108dd7a1a2032cd905e1cd8b421ef46c3807: Status 404 returned error can't find the container with id 685e8f13695bf51f37b393d593c7108dd7a1a2032cd905e1cd8b421ef46c3807
Jul 02 18:30:09.690185 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.690161    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7wdxk"
Jul 02 18:30:09.698459 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:09.698432    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72fc1e32_b32e_415f_9101_62498e24ea7b.slice/crio-b98eb5438931bb3f1a385d19df0f6053a6890f7f268584e2bc604b66fbad8442 WatchSource:0}: Error finding container b98eb5438931bb3f1a385d19df0f6053a6890f7f268584e2bc604b66fbad8442: Status 404 returned error can't find the container with id b98eb5438931bb3f1a385d19df0f6053a6890f7f268584e2bc604b66fbad8442
Jul 02 18:30:09.707648 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.707625    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc"
Jul 02 18:30:09.716001 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:09.715978    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c1fe04_027c_473f_ad0a_0cabf800d9e3.slice/crio-652033322c8032466382da3f95cd3a47c7d9bb1f609393eb583a697452a52bd6 WatchSource:0}: Error finding container 652033322c8032466382da3f95cd3a47c7d9bb1f609393eb583a697452a52bd6: Status 404 returned error can't find the container with id 652033322c8032466382da3f95cd3a47c7d9bb1f609393eb583a697452a52bd6
Jul 02 18:30:09.723301 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.723256    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6mtkp"
Jul 02 18:30:09.730606 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:09.730581    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a96438_0081_4f62_aa03_cd7cf1bb0f2c.slice/crio-2d67305b3c834f3bc16e26b716a725aa8aef34899bf3fafd96bea981d4e6f218 WatchSource:0}: Error finding container 2d67305b3c834f3bc16e26b716a725aa8aef34899bf3fafd96bea981d4e6f218: Status 404 returned error can't find the container with id 2d67305b3c834f3bc16e26b716a725aa8aef34899bf3fafd96bea981d4e6f218
Jul 02 18:30:09.737061 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.737044    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6cwjc"
Jul 02 18:30:09.744574 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:09.744550    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02cf5868_e870_47eb_870f_c87dcabdae5a.slice/crio-2f18b025d00ef4260e7fe44504d2b176c722559a56b8f10a38c8aa25999d2693 WatchSource:0}: Error finding container 2f18b025d00ef4260e7fe44504d2b176c722559a56b8f10a38c8aa25999d2693: Status 404 returned error can't find the container with id 2f18b025d00ef4260e7fe44504d2b176c722559a56b8f10a38c8aa25999d2693
Jul 02 18:30:09.753927 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.753910    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6f6sb"
Jul 02 18:30:09.762748 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:09.762729    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e2289a_1242_48fa_9b3e_a89b46c2c9d8.slice/crio-afc614e0ec1f14bd322f7c133082ce113efee2322eb8cd4dbb6cae9fd1f577ac WatchSource:0}: Error finding container afc614e0ec1f14bd322f7c133082ce113efee2322eb8cd4dbb6cae9fd1f577ac: Status 404 returned error can't find the container with id afc614e0ec1f14bd322f7c133082ce113efee2322eb8cd4dbb6cae9fd1f577ac
Jul 02 18:30:09.769871 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.769853    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hhvg5"
Jul 02 18:30:09.775282 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.775254    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rglzr"
Jul 02 18:30:09.777079 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:09.777060    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01f69e9_cff9_4c5a_97d0_68a53d05b20b.slice/crio-9206fddc4decf6487c264ca4a13b19a2bf8db267b5d88a9548d127fc0629df16 WatchSource:0}: Error finding container 9206fddc4decf6487c264ca4a13b19a2bf8db267b5d88a9548d127fc0629df16: Status 404 returned error can't find the container with id 9206fddc4decf6487c264ca4a13b19a2bf8db267b5d88a9548d127fc0629df16
Jul 02 18:30:09.783012 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:09.782993    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84ffa74_4554_441b_8e99_be4646050843.slice/crio-af6e32723c80c2d48be649bfe436048afff74d1c4d13577ae365cfece9b80c74 WatchSource:0}: Error finding container af6e32723c80c2d48be649bfe436048afff74d1c4d13577ae365cfece9b80c74: Status 404 returned error can't find the container with id af6e32723c80c2d48be649bfe436048afff74d1c4d13577ae365cfece9b80c74
Jul 02 18:30:09.882964 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:09.882935    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 18:30:10.041801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.041712    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs\") pod \"network-metrics-daemon-m2h4c\" (UID: \"780da6ef-b5e6-43c9-b539-185cc5434c63\") " pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:10.041801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.041776    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:10.042045 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:10.041975    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:10.042107 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:10.042047    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret podName:d3682ee4-82ef-4457-a048-4de940c25ac0 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:11.042025838 +0000 UTC m=+3.111653347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret") pod "global-pull-secret-syncer-5j4fb" (UID: "d3682ee4-82ef-4457-a048-4de940c25ac0") : object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:10.042603 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:10.042501    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:10.042603 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:10.042567    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs podName:780da6ef-b5e6-43c9-b539-185cc5434c63 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:11.04254828 +0000 UTC m=+3.112175795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs") pod "network-metrics-daemon-m2h4c" (UID: "780da6ef-b5e6-43c9-b539-185cc5434c63") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:10.142643 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.142542    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwj6\" (UniqueName: \"kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6\") pod \"network-check-target-b7jnx\" (UID: \"7340eb6c-cda8-42dc-9abe-2b16be84f537\") " pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:10.142852 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:10.142764    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 18:30:10.142852 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:10.142796    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 18:30:10.142852 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:10.142811    2575 projected.go:196] Error preparing data for projected volume kube-api-access-sxwj6 for pod openshift-network-diagnostics/network-check-target-b7jnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:10.143054 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:10.142881    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6 podName:7340eb6c-cda8-42dc-9abe-2b16be84f537 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:11.142857153 +0000 UTC m=+3.212484777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxwj6" (UniqueName: "kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6") pod "network-check-target-b7jnx" (UID: "7340eb6c-cda8-42dc-9abe-2b16be84f537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:10.373061 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.372454    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 18:25:09 +0000 UTC" deadline="2028-04-19 08:19:51.57539278 +0000 UTC"
Jul 02 18:30:10.373061 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.372519    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15757h49m41.202878215s"
Jul 02 18:30:10.483979 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.483939    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:10.484141 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:10.484088    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:10.484642 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.484617    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:10.484813 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:10.484736    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:10.514739 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.514622    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rglzr" event={"ID":"e84ffa74-4554-441b-8e99-be4646050843","Type":"ContainerStarted","Data":"af6e32723c80c2d48be649bfe436048afff74d1c4d13577ae365cfece9b80c74"}
Jul 02 18:30:10.525362 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.525331    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hhvg5" event={"ID":"c01f69e9-cff9-4c5a-97d0-68a53d05b20b","Type":"ContainerStarted","Data":"9206fddc4decf6487c264ca4a13b19a2bf8db267b5d88a9548d127fc0629df16"}
Jul 02 18:30:10.538793 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.538756    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 18:30:10.560890 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.560851    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6sb" event={"ID":"98e2289a-1242-48fa-9b3e-a89b46c2c9d8","Type":"ContainerStarted","Data":"afc614e0ec1f14bd322f7c133082ce113efee2322eb8cd4dbb6cae9fd1f577ac"}
Jul 02 18:30:10.569846 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.569797    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6cwjc" event={"ID":"02cf5868-e870-47eb-870f-c87dcabdae5a","Type":"ContainerStarted","Data":"2f18b025d00ef4260e7fe44504d2b176c722559a56b8f10a38c8aa25999d2693"}
Jul 02 18:30:10.594189 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.594146    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6mtkp" event={"ID":"d5a96438-0081-4f62-aa03-cd7cf1bb0f2c","Type":"ContainerStarted","Data":"2d67305b3c834f3bc16e26b716a725aa8aef34899bf3fafd96bea981d4e6f218"}
Jul 02 18:30:10.615112 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.615076    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc" event={"ID":"a0c1fe04-027c-473f-ad0a-0cabf800d9e3","Type":"ContainerStarted","Data":"652033322c8032466382da3f95cd3a47c7d9bb1f609393eb583a697452a52bd6"}
Jul 02 18:30:10.638880 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.638769    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal" event={"ID":"000fc49438e162666c8381fedce28b6d","Type":"ContainerStarted","Data":"5b77f8a8b65cdfd62f33ecce3c13dd4694000e4995b607bd083dd1ac7891fb01"}
Jul 02 18:30:10.665691 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.665646    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal" event={"ID":"06004644724e4a334662d055fee9ec18","Type":"ContainerStarted","Data":"3cea103836085c1e3e42b24f0d4db16479cbfcfabdff6cbf5ca8b6809e39ca65"}
Jul 02 18:30:10.684288 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.680513    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7wdxk" event={"ID":"72fc1e32-b32e-415f-9101-62498e24ea7b","Type":"ContainerStarted","Data":"b98eb5438931bb3f1a385d19df0f6053a6890f7f268584e2bc604b66fbad8442"}
Jul 02 18:30:10.716768 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.716727    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4" event={"ID":"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b","Type":"ContainerStarted","Data":"685e8f13695bf51f37b393d593c7108dd7a1a2032cd905e1cd8b421ef46c3807"}
Jul 02 18:30:10.734015 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:10.733976    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m8sht" event={"ID":"f1197266-26d4-4a4d-9652-0db22827bf3b","Type":"ContainerStarted","Data":"df47527ecc6de901166c1182f08b12cbb7119ed27136bd353731389c7e93f78e"}
Jul 02 18:30:11.050161 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:11.050127    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs\") pod \"network-metrics-daemon-m2h4c\" (UID: \"780da6ef-b5e6-43c9-b539-185cc5434c63\") " pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:11.050353 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:11.050187    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:11.050353 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:11.050344    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:11.050488 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:11.050405    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret podName:d3682ee4-82ef-4457-a048-4de940c25ac0 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:13.0503852 +0000 UTC m=+5.120012712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret") pod "global-pull-secret-syncer-5j4fb" (UID: "d3682ee4-82ef-4457-a048-4de940c25ac0") : object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:11.050818 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:11.050794    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:11.050919 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:11.050850    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs podName:780da6ef-b5e6-43c9-b539-185cc5434c63 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:13.050831869 +0000 UTC m=+5.120459382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs") pod "network-metrics-daemon-m2h4c" (UID: "780da6ef-b5e6-43c9-b539-185cc5434c63") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:11.151165 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:11.150520    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwj6\" (UniqueName: \"kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6\") pod \"network-check-target-b7jnx\" (UID: \"7340eb6c-cda8-42dc-9abe-2b16be84f537\") " pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:11.151165 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:11.150715    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 18:30:11.151165 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:11.150736    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 18:30:11.151165 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:11.150749    2575 projected.go:196] Error preparing data for projected volume kube-api-access-sxwj6 for pod openshift-network-diagnostics/network-check-target-b7jnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:11.151165 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:11.150815    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6 podName:7340eb6c-cda8-42dc-9abe-2b16be84f537 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:13.150795284 +0000 UTC m=+5.220422815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxwj6" (UniqueName: "kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6") pod "network-check-target-b7jnx" (UID: "7340eb6c-cda8-42dc-9abe-2b16be84f537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:11.373622 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:11.373533    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 18:25:09 +0000 UTC" deadline="2027-12-13 10:51:22.929391282 +0000 UTC"
Jul 02 18:30:11.373622 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:11.373573    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12688h21m11.555822659s"
Jul 02 18:30:11.483747 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:11.483716    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:11.483948 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:11.483864    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:11.703063 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:11.702959    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 18:30:12.483901 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:12.483076    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:12.483901 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:12.483236    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:12.483901 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:12.483745    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:12.483901 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:12.483855    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:13.067497 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:13.067455    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs\") pod \"network-metrics-daemon-m2h4c\" (UID: \"780da6ef-b5e6-43c9-b539-185cc5434c63\") " pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:13.067681 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:13.067515    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:13.067783 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:13.067705    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:13.067783 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:13.067774    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret podName:d3682ee4-82ef-4457-a048-4de940c25ac0 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:17.067752488 +0000 UTC m=+9.137379999 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret") pod "global-pull-secret-syncer-5j4fb" (UID: "d3682ee4-82ef-4457-a048-4de940c25ac0") : object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:13.068248 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:13.068227    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:13.068366 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:13.068304    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs podName:780da6ef-b5e6-43c9-b539-185cc5434c63 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:17.068286494 +0000 UTC m=+9.137914017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs") pod "network-metrics-daemon-m2h4c" (UID: "780da6ef-b5e6-43c9-b539-185cc5434c63") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:13.168039 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:13.167948    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwj6\" (UniqueName: \"kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6\") pod \"network-check-target-b7jnx\" (UID: \"7340eb6c-cda8-42dc-9abe-2b16be84f537\") " pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:13.168228 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:13.168129    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 18:30:13.168228 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:13.168154    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 18:30:13.168228 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:13.168175    2575 projected.go:196] Error preparing data for projected volume kube-api-access-sxwj6 for pod openshift-network-diagnostics/network-check-target-b7jnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:13.168426 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:13.168249    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6 podName:7340eb6c-cda8-42dc-9abe-2b16be84f537 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:17.16821622 +0000 UTC m=+9.237843743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxwj6" (UniqueName: "kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6") pod "network-check-target-b7jnx" (UID: "7340eb6c-cda8-42dc-9abe-2b16be84f537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:13.484085 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:13.483490    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:13.484085 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:13.483635    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:14.484302 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:14.484123    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:14.484302 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:14.484300    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:14.485256 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:14.485005    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:14.485256 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:14.485155    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:15.483642 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:15.483605    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:15.483818 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:15.483787    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:16.483970 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:16.483936    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:16.484515 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:16.484076    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:16.484515 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:16.484130    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:16.484515 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:16.484269    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:17.101611 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:17.101304    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs\") pod \"network-metrics-daemon-m2h4c\" (UID: \"780da6ef-b5e6-43c9-b539-185cc5434c63\") " pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:17.101611 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:17.101589    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:17.101832 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:17.101481    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:17.101891 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:17.101839    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs podName:780da6ef-b5e6-43c9-b539-185cc5434c63 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:25.101818013 +0000 UTC m=+17.171445527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs") pod "network-metrics-daemon-m2h4c" (UID: "780da6ef-b5e6-43c9-b539-185cc5434c63") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:17.101891 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:17.101773    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:17.101998 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:17.101915    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret podName:d3682ee4-82ef-4457-a048-4de940c25ac0 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:25.101902722 +0000 UTC m=+17.171530243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret") pod "global-pull-secret-syncer-5j4fb" (UID: "d3682ee4-82ef-4457-a048-4de940c25ac0") : object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:17.202609 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:17.202569    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwj6\" (UniqueName: \"kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6\") pod \"network-check-target-b7jnx\" (UID: \"7340eb6c-cda8-42dc-9abe-2b16be84f537\") " pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:17.202789 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:17.202750    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 18:30:17.202789 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:17.202773    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 18:30:17.202789 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:17.202786    2575 projected.go:196] Error preparing data for projected volume kube-api-access-sxwj6 for pod openshift-network-diagnostics/network-check-target-b7jnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:17.202958 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:17.202850    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6 podName:7340eb6c-cda8-42dc-9abe-2b16be84f537 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:25.202830035 +0000 UTC m=+17.272457542 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxwj6" (UniqueName: "kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6") pod "network-check-target-b7jnx" (UID: "7340eb6c-cda8-42dc-9abe-2b16be84f537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:17.483655 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:17.483584    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:17.483800 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:17.483715    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:18.485333 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:18.484722    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:18.485333 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:18.484840    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:18.485333 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:18.485186    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:18.485333 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:18.485289    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:19.483777 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:19.483739    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:19.483956 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:19.483875    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:20.483389 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:20.483352    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:20.483822 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:20.483474    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:20.483822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:20.483537    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:20.483822 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:20.483648    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:21.483879 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:21.483840    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:21.484357 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:21.483967    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:22.483643 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:22.483607    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:22.483808 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:22.483750    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:22.483871 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:22.483806    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:22.483956 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:22.483933    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:23.483625 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:23.483589    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:23.483811 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:23.483711    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:24.483984 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:24.483937    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:24.484473 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:24.484062    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:24.484473 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:24.484100    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:24.484473 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:24.484161    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:25.162306 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:25.162267    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs\") pod \"network-metrics-daemon-m2h4c\" (UID: \"780da6ef-b5e6-43c9-b539-185cc5434c63\") " pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:25.162306 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:25.162310    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:25.162518 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:25.162429    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:25.162518 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:25.162490    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs podName:780da6ef-b5e6-43c9-b539-185cc5434c63 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:41.162475369 +0000 UTC m=+33.232102879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs") pod "network-metrics-daemon-m2h4c" (UID: "780da6ef-b5e6-43c9-b539-185cc5434c63") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:25.162589 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:25.162430    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:25.162589 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:25.162564    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret podName:d3682ee4-82ef-4457-a048-4de940c25ac0 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:41.162551861 +0000 UTC m=+33.232179367 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret") pod "global-pull-secret-syncer-5j4fb" (UID: "d3682ee4-82ef-4457-a048-4de940c25ac0") : object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:25.263309 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:25.263272    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwj6\" (UniqueName: \"kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6\") pod \"network-check-target-b7jnx\" (UID: \"7340eb6c-cda8-42dc-9abe-2b16be84f537\") " pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:25.263473 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:25.263442    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 18:30:25.263473 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:25.263464    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 18:30:25.263473 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:25.263474    2575 projected.go:196] Error preparing data for projected volume kube-api-access-sxwj6 for pod openshift-network-diagnostics/network-check-target-b7jnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:25.263586 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:25.263562    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6 podName:7340eb6c-cda8-42dc-9abe-2b16be84f537 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:41.263544069 +0000 UTC m=+33.333171589 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxwj6" (UniqueName: "kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6") pod "network-check-target-b7jnx" (UID: "7340eb6c-cda8-42dc-9abe-2b16be84f537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:25.483932 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:25.483860    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:25.484076 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:25.483973    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:26.482898 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:26.482868    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:26.482898 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:26.482895    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:26.483211 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:26.483003    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:26.483211 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:26.483113    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:27.483309 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:27.483277    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:27.483764 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:27.483399    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:28.484167 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:28.484126    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:28.484658 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:28.484251    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:28.484658 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:28.484330    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:28.484658 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:28.484460    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:29.483843 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:29.483809    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:29.484000 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:29.483933    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:30.483206 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.482960    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:30.484024 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:30.483309    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:30.484024 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.482974    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:30.484024 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:30.483758    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:30.789853 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.789734    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal" event={"ID":"000fc49438e162666c8381fedce28b6d","Type":"ContainerStarted","Data":"93c36c7790d643e94238a7b4a4ceccd718e8e85087e0bc1ce72f36b5a5d4199b"}
Jul 02 18:30:30.791280 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.791254    2575 generic.go:356] "Generic (PLEG): container finished" podID="06004644724e4a334662d055fee9ec18" containerID="a3621a2ca9efd0c6676e5dc5f723cc1ba724241efb98b94962b72e1eb7522171" exitCode=0
Jul 02 18:30:30.791399 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.791344    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal" event={"ID":"06004644724e4a334662d055fee9ec18","Type":"ContainerDied","Data":"a3621a2ca9efd0c6676e5dc5f723cc1ba724241efb98b94962b72e1eb7522171"}
Jul 02 18:30:30.792669 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.792644    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7wdxk" event={"ID":"72fc1e32-b32e-415f-9101-62498e24ea7b","Type":"ContainerStarted","Data":"a0b15c127d803971b48cdc55cc5c1eae95b38143a613c8d885cf9a1dd0b9c3fe"}
Jul 02 18:30:30.795372 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.795354    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 18:30:30.795684 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.795665    2575 generic.go:356] "Generic (PLEG): container finished" podID="0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b" containerID="35716f68cc7a93bb061af4cb108bac0836c97299c3eda581eea12d1e3cc3c1c5" exitCode=1
Jul 02 18:30:30.795755 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.795732    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4" event={"ID":"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b","Type":"ContainerStarted","Data":"12878df47a54d33eea5115a6e960f98cd78b39791646acab8a50dced957e4222"}
Jul 02 18:30:30.795824 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.795806    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4" event={"ID":"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b","Type":"ContainerStarted","Data":"9a376962f3f87a2339f1dfc0789169021dc40f8c6fdaf813f65b9e8152085173"}
Jul 02 18:30:30.795877 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.795863    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4" event={"ID":"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b","Type":"ContainerStarted","Data":"89daff563082109e588d15fd35cf25257c58cf614ef6c1a1ff6f5df408cba4ff"}
Jul 02 18:30:30.795940 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.795928    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4" event={"ID":"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b","Type":"ContainerStarted","Data":"9edc03aaf00711e0dede127ec478fe0c7ce5eb85677424635b6fe31f0b10dda8"}
Jul 02 18:30:30.795980 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.795970    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4" event={"ID":"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b","Type":"ContainerDied","Data":"35716f68cc7a93bb061af4cb108bac0836c97299c3eda581eea12d1e3cc3c1c5"}
Jul 02 18:30:30.796018 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.796010    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4" event={"ID":"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b","Type":"ContainerStarted","Data":"dcd87e3dff6f56021e85045ebe3986e0ac12c580ec7459d17e08b4fda1e6a447"}
Jul 02 18:30:30.797142 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.797124    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rglzr" event={"ID":"e84ffa74-4554-441b-8e99-be4646050843","Type":"ContainerStarted","Data":"517045c53a33c272efc9f41dc2b78c86d6c8203d33ae8ef370ec96ae45714b5a"}
Jul 02 18:30:30.798468 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.798448    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hhvg5" event={"ID":"c01f69e9-cff9-4c5a-97d0-68a53d05b20b","Type":"ContainerStarted","Data":"a47e2109914e0cee11cdf91753559298525b49caadc1bc213cdc2566a10b9a5c"}
Jul 02 18:30:30.799781 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.799759    2575 generic.go:356] "Generic (PLEG): container finished" podID="98e2289a-1242-48fa-9b3e-a89b46c2c9d8" containerID="5348012a73e4a32d9f19511220a6fd1c82cde7b0020a02c0f068a5bb462b7479" exitCode=0
Jul 02 18:30:30.799869 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.799830    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6sb" event={"ID":"98e2289a-1242-48fa-9b3e-a89b46c2c9d8","Type":"ContainerDied","Data":"5348012a73e4a32d9f19511220a6fd1c82cde7b0020a02c0f068a5bb462b7479"}
Jul 02 18:30:30.801257 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.801141    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6cwjc" event={"ID":"02cf5868-e870-47eb-870f-c87dcabdae5a","Type":"ContainerStarted","Data":"17b4a3f233ff2328b601d51b54ed3ea88c0e935e308b8991f172c0e063078787"}
Jul 02 18:30:30.802488 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.802453    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6mtkp" event={"ID":"d5a96438-0081-4f62-aa03-cd7cf1bb0f2c","Type":"ContainerStarted","Data":"94bb87a62feb376d7dc7366a8ff2abee9536435d2d369c9b90eb11ef4e1b994d"}
Jul 02 18:30:30.803859 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.803838    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc" event={"ID":"a0c1fe04-027c-473f-ad0a-0cabf800d9e3","Type":"ContainerStarted","Data":"807f5f0eabfb1d2ceb56f34441c52bcb3f67525c6ae9ff150b0a3564c6d16f44"}
Jul 02 18:30:30.805403 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.805365    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal" podStartSLOduration=21.805353947 podStartE2EDuration="21.805353947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:30.805029446 +0000 UTC m=+22.874656974" watchObservedRunningTime="2026-07-02 18:30:30.805353947 +0000 UTC m=+22.874981475"
Jul 02 18:30:30.821134 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.821065    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7wdxk" podStartSLOduration=1.901450941 podStartE2EDuration="21.82104324s" totalImagesPullingTime="19.919592299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:30.82017326 +0000 UTC m=+22.889800789" watchObservedRunningTime="2026-07-02 18:30:30.82104324 +0000 UTC m=+22.890670769"
Jul 02 18:30:30.852005 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.851935    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6mtkp" podStartSLOduration=1.964476251 podStartE2EDuration="21.85191303s" totalImagesPullingTime="19.887436779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:30.851322283 +0000 UTC m=+22.920949835" watchObservedRunningTime="2026-07-02 18:30:30.85191303 +0000 UTC m=+22.921540559"
Jul 02 18:30:30.872751 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.872704    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6cwjc" podStartSLOduration=1.944570344 podStartE2EDuration="21.872685622s" totalImagesPullingTime="19.928115278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:30.872505979 +0000 UTC m=+22.942133526" watchObservedRunningTime="2026-07-02 18:30:30.872685622 +0000 UTC m=+22.942313152"
Jul 02 18:30:30.897023 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.896992    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7wdxk"
Jul 02 18:30:30.914776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.914732    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hhvg5" podStartSLOduration=2.058824249 podStartE2EDuration="21.914716071s" totalImagesPullingTime="19.855891822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:30.91438675 +0000 UTC m=+22.984014277" watchObservedRunningTime="2026-07-02 18:30:30.914716071 +0000 UTC m=+22.984343598"
Jul 02 18:30:30.933805 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:30.933751    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rglzr" podStartSLOduration=2.040533451 podStartE2EDuration="21.933734737s" totalImagesPullingTime="19.893201286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:30.93347106 +0000 UTC m=+23.003098589" watchObservedRunningTime="2026-07-02 18:30:30.933734737 +0000 UTC m=+23.003362264"
Jul 02 18:30:31.483647 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:31.483622    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:31.484168 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:31.483730    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:31.792936 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:31.792904    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 18:30:31.808850 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:31.808804    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc" event={"ID":"a0c1fe04-027c-473f-ad0a-0cabf800d9e3","Type":"ContainerStarted","Data":"13571e49a48702debf0801bfee36295df35f27fed6b9eaa3fa51e90c5abcacca"}
Jul 02 18:30:31.810721 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:31.810666    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal" event={"ID":"06004644724e4a334662d055fee9ec18","Type":"ContainerStarted","Data":"0f1a338e2c91df63ae11bbcd69eeed58d9a3ae5cae56c6587e508568609b6b4c"}
Jul 02 18:30:31.812483 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:31.812415    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m8sht" event={"ID":"f1197266-26d4-4a4d-9652-0db22827bf3b","Type":"ContainerStarted","Data":"dea883e1912e0ba8963c9a3e7c347d1a7194adf564b29abd2ae328537304dcd1"}
Jul 02 18:30:31.828750 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:31.828699    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal" podStartSLOduration=22.828678546 podStartE2EDuration="22.828678546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:31.828586897 +0000 UTC m=+23.898214425" watchObservedRunningTime="2026-07-02 18:30:31.828678546 +0000 UTC m=+23.898306076"
Jul 02 18:30:31.844475 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:31.844421    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m8sht" podStartSLOduration=2.833866538 podStartE2EDuration="22.844402483s" totalImagesPullingTime="20.010535945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:31.84396911 +0000 UTC m=+23.913596667" watchObservedRunningTime="2026-07-02 18:30:31.844402483 +0000 UTC m=+23.914030010"
Jul 02 18:30:32.400761 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:32.400636    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T18:30:31.792924327Z","UUID":"16324845-3063-4941-835b-8bb4d3ffcfe1","Handler":null,"Name":"","Endpoint":""}
Jul 02 18:30:32.402668 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:32.402516    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 18:30:32.402668 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:32.402567    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 18:30:32.484039 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:32.484009    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:32.484572 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:32.484142    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:32.484572 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:32.484008    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:32.484572 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:32.484294    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:32.818135 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:32.818110    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 18:30:32.818600 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:32.818563    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4" event={"ID":"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b","Type":"ContainerStarted","Data":"6d41779aa24e521ff48e0d6a4a0503f62b009e8b9b84ca3abf39559f67913845"}
Jul 02 18:30:33.483405 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:33.483369    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:33.483685 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:33.483497    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:33.796051 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:33.796015    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7wdxk"
Jul 02 18:30:33.796668 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:33.796641    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7wdxk"
Jul 02 18:30:33.823053 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:33.823014    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc" event={"ID":"a0c1fe04-027c-473f-ad0a-0cabf800d9e3","Type":"ContainerStarted","Data":"0c80e50072b9c3285b04a983f58034f8b66a05426074a425ea9a2df8ee256146"}
Jul 02 18:30:33.823838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:33.823812    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7wdxk"
Jul 02 18:30:33.840961 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:33.840908    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48csc" podStartSLOduration=2.571594807 podStartE2EDuration="24.840891403s" totalImagesPullingTime="22.269296596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:33.84032516 +0000 UTC m=+25.909952702" watchObservedRunningTime="2026-07-02 18:30:33.840891403 +0000 UTC m=+25.910518930"
Jul 02 18:30:34.483162 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:34.483125    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:34.483338 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:34.483288    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:34.483396 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:34.483337    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:34.483527 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:34.483451    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:35.483508 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:35.483289    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:35.484138 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:35.483545    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:35.830625 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:35.830593    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 18:30:35.830988 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:35.830954    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4" event={"ID":"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b","Type":"ContainerStarted","Data":"042c06ec5c52153f406a4029bf895692fdef7fed73dbd58c9ee83a509d27b77a"}
Jul 02 18:30:35.831368 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:35.831297    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:35.831368 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:35.831358    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:35.831545 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:35.831404    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:35.831545 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:35.831502    2575 scope.go:117] "RemoveContainer" containerID="35716f68cc7a93bb061af4cb108bac0836c97299c3eda581eea12d1e3cc3c1c5"
Jul 02 18:30:35.833049 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:35.833022    2575 generic.go:356] "Generic (PLEG): container finished" podID="98e2289a-1242-48fa-9b3e-a89b46c2c9d8" containerID="e9a3da06764dbf8270a44360a8c80f308cc9df7e401a61039a780e1c0ad5d212" exitCode=0
Jul 02 18:30:35.833157 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:35.833105    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6sb" event={"ID":"98e2289a-1242-48fa-9b3e-a89b46c2c9d8","Type":"ContainerDied","Data":"e9a3da06764dbf8270a44360a8c80f308cc9df7e401a61039a780e1c0ad5d212"}
Jul 02 18:30:35.849498 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:35.849478    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:35.849685 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:35.849665    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:30:36.483807 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:36.483555    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:36.484290 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:36.483608    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:36.484290 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:36.483931    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:36.484290 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:36.483958    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:36.811831 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:36.811775    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m2h4c"]
Jul 02 18:30:36.815482 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:36.815453    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b7jnx"]
Jul 02 18:30:36.816233 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:36.816186    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5j4fb"]
Jul 02 18:30:36.816346 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:36.816333    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:36.816460 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:36.816443    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:36.838241 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:36.838187    2575 generic.go:356] "Generic (PLEG): container finished" podID="98e2289a-1242-48fa-9b3e-a89b46c2c9d8" containerID="9ce4197e27b1a64adf03be622b1c8d6ae613bd2e01bbacc358dd49404d396fd4" exitCode=0
Jul 02 18:30:36.838241 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:36.838241    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6sb" event={"ID":"98e2289a-1242-48fa-9b3e-a89b46c2c9d8","Type":"ContainerDied","Data":"9ce4197e27b1a64adf03be622b1c8d6ae613bd2e01bbacc358dd49404d396fd4"}
Jul 02 18:30:36.842664 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:36.842647    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 18:30:36.843017 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:36.842997    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4" event={"ID":"0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b","Type":"ContainerStarted","Data":"323598f512740968884467d63ce813be8b4a52b5e099676c0a13aa628a0ebcbe"}
Jul 02 18:30:36.843109 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:36.843024    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:36.843109 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:36.843036    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:36.843218 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:36.843114    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:36.843407 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:36.843385    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:36.890563 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:36.890517    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4" podStartSLOduration=7.7806582330000005 podStartE2EDuration="27.890499786s" totalImagesPullingTime="20.109841553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:36.88770993 +0000 UTC m=+28.957337449" watchObservedRunningTime="2026-07-02 18:30:36.890499786 +0000 UTC m=+28.960127315"
Jul 02 18:30:38.486013 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:38.485981    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:38.486013 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:38.486015    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:38.486556 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:38.486073    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:38.486556 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:38.486086    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:38.486556 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:38.486165    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:38.486556 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:38.486234    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:38.850234 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:38.850182    2575 generic.go:356] "Generic (PLEG): container finished" podID="98e2289a-1242-48fa-9b3e-a89b46c2c9d8" containerID="8f746a7381256c9b4e62aa92dce404e919d7354333c47cd92813d4ab401683fa" exitCode=0
Jul 02 18:30:38.850423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:38.850270    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6sb" event={"ID":"98e2289a-1242-48fa-9b3e-a89b46c2c9d8","Type":"ContainerDied","Data":"8f746a7381256c9b4e62aa92dce404e919d7354333c47cd92813d4ab401683fa"}
Jul 02 18:30:40.483500 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:40.483162    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:40.483500 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:40.483226    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:40.483500 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:40.483164    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:40.483500 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:40.483313    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:40.483500 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:40.483420    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:40.483500 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:40.483500    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:41.178510 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:41.178468    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs\") pod \"network-metrics-daemon-m2h4c\" (UID: \"780da6ef-b5e6-43c9-b539-185cc5434c63\") " pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:41.178701 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:41.178605    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:41.178701 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:41.178616    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:41.178701 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:41.178674    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs podName:780da6ef-b5e6-43c9-b539-185cc5434c63 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:13.178656669 +0000 UTC m=+65.248284177 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs") pod "network-metrics-daemon-m2h4c" (UID: "780da6ef-b5e6-43c9-b539-185cc5434c63") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:41.178946 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:41.178721    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:41.178946 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:41.178783    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret podName:d3682ee4-82ef-4457-a048-4de940c25ac0 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:13.178764999 +0000 UTC m=+65.248392519 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret") pod "global-pull-secret-syncer-5j4fb" (UID: "d3682ee4-82ef-4457-a048-4de940c25ac0") : object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:41.279912 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:41.279882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwj6\" (UniqueName: \"kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6\") pod \"network-check-target-b7jnx\" (UID: \"7340eb6c-cda8-42dc-9abe-2b16be84f537\") " pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:41.280080 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:41.279997    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 18:30:41.280080 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:41.280009    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 18:30:41.280080 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:41.280019    2575 projected.go:196] Error preparing data for projected volume kube-api-access-sxwj6 for pod openshift-network-diagnostics/network-check-target-b7jnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:41.280080 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:41.280064    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6 podName:7340eb6c-cda8-42dc-9abe-2b16be84f537 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:13.280051268 +0000 UTC m=+65.349678776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxwj6" (UniqueName: "kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6") pod "network-check-target-b7jnx" (UID: "7340eb6c-cda8-42dc-9abe-2b16be84f537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:42.484056 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:42.483858    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:42.484543 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:42.484144    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j4fb" podUID="d3682ee4-82ef-4457-a048-4de940c25ac0"
Jul 02 18:30:42.484543 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:42.483898    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:42.484543 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:42.484256    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7jnx" podUID="7340eb6c-cda8-42dc-9abe-2b16be84f537"
Jul 02 18:30:42.484543 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:42.483869    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:42.484543 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:42.484336    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2h4c" podUID="780da6ef-b5e6-43c9-b539-185cc5434c63"
Jul 02 18:30:43.672893 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.672864    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeReady"
Jul 02 18:30:43.673288 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.672990    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 18:30:43.712511 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.712480    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-596cd447fb-jh626"]
Jul 02 18:30:43.714435 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.714409    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.719344 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.719315    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 02 18:30:43.719614 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.719599    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 02 18:30:43.719679 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.719663    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2gz4h\""
Jul 02 18:30:43.719830 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.719806    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 02 18:30:43.725881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.725861    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9zq5n"]
Jul 02 18:30:43.727521 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.727505    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:43.727595 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.727570    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 02 18:30:43.729496 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.729477    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-596cd447fb-jh626"]
Jul 02 18:30:43.730467 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.730450    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 18:30:43.730568 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.730486    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cl9vj\""
Jul 02 18:30:43.730620 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.730612    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 18:30:43.745560 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.745539    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9zq5n"]
Jul 02 18:30:43.749647 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.749625    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:43.767698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.767676    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qt4zw"]
Jul 02 18:30:43.769451 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.769437    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qt4zw"
Jul 02 18:30:43.772519 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.772502    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2hlqm\""
Jul 02 18:30:43.772715 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.772694    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 18:30:43.772782 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.772697    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 18:30:43.772838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.772824    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 18:30:43.779326 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.779305    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qt4zw"]
Jul 02 18:30:43.795994 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.795967    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-bound-sa-token\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.796090 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.796002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed77eb69-1101-464b-9a74-e14bb394b7fa-image-registry-private-configuration\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.796090 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.796020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed77eb69-1101-464b-9a74-e14bb394b7fa-ca-trust-extracted\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.796163 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.796096    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed77eb69-1101-464b-9a74-e14bb394b7fa-installation-pull-secrets\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.796163 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.796124    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hbhp\" (UniqueName: \"kubernetes.io/projected/684a7ad0-005b-49d6-817b-b5bc7cbfb719-kube-api-access-4hbhp\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:43.796163 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.796146    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.796289 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.796160    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed77eb69-1101-464b-9a74-e14bb394b7fa-trusted-ca\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.796289 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.796176    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/684a7ad0-005b-49d6-817b-b5bc7cbfb719-config-volume\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:43.796289 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.796206    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:43.796289 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.796282    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-certificates\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.796439 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.796314    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58cq7\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-kube-api-access-58cq7\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.796439 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.796339    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684a7ad0-005b-49d6-817b-b5bc7cbfb719-tmp-dir\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:43.843616 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.843591    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:43.864293 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.864266    2575 generic.go:356] "Generic (PLEG): container finished" podID="98e2289a-1242-48fa-9b3e-a89b46c2c9d8" containerID="b80de96aa1ea04e8c299a7ad94f32292dca20586b22ae3031b4d46fcc9add7e3" exitCode=0
Jul 02 18:30:43.864383 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.864299    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6sb" event={"ID":"98e2289a-1242-48fa-9b3e-a89b46c2c9d8","Type":"ContainerDied","Data":"b80de96aa1ea04e8c299a7ad94f32292dca20586b22ae3031b4d46fcc9add7e3"}
Jul 02 18:30:43.897359 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.897315    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.897359 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.897359    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed77eb69-1101-464b-9a74-e14bb394b7fa-trusted-ca\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.897538 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.897386    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/684a7ad0-005b-49d6-817b-b5bc7cbfb719-config-volume\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:43.897538 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.897408    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:43.897538 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.897440    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-certificates\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.897538 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.897470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58cq7\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-kube-api-access-58cq7\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.897538 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.897494    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684a7ad0-005b-49d6-817b-b5bc7cbfb719-tmp-dir\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:43.897538 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.897529    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cstd6\" (UniqueName: \"kubernetes.io/projected/6fa3f945-1298-44c4-8ec7-33014cddcd23-kube-api-access-cstd6\") pod \"ingress-canary-qt4zw\" (UID: \"6fa3f945-1298-44c4-8ec7-33014cddcd23\") " pod="openshift-ingress-canary/ingress-canary-qt4zw"
Jul 02 18:30:43.897826 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.897578    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-bound-sa-token\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.897826 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.897615    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed77eb69-1101-464b-9a74-e14bb394b7fa-image-registry-private-configuration\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.897826 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.897641    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed77eb69-1101-464b-9a74-e14bb394b7fa-ca-trust-extracted\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.897826 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.897664    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fa3f945-1298-44c4-8ec7-33014cddcd23-cert\") pod \"ingress-canary-qt4zw\" (UID: \"6fa3f945-1298-44c4-8ec7-33014cddcd23\") " pod="openshift-ingress-canary/ingress-canary-qt4zw"
Jul 02 18:30:43.898106 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:43.898087    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 18:30:43.898209 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.898112    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684a7ad0-005b-49d6-817b-b5bc7cbfb719-tmp-dir\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:43.898349 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.898323    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-certificates\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.898418 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.898396    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/684a7ad0-005b-49d6-817b-b5bc7cbfb719-config-volume\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:43.898483 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.898421    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed77eb69-1101-464b-9a74-e14bb394b7fa-installation-pull-secrets\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.898483 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.898451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hbhp\" (UniqueName: \"kubernetes.io/projected/684a7ad0-005b-49d6-817b-b5bc7cbfb719-kube-api-access-4hbhp\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:43.898586 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:43.898491    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 18:30:43.898586 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:43.898505    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596cd447fb-jh626: secret "image-registry-tls" not found
Jul 02 18:30:43.898586 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:43.898568    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls podName:ed77eb69-1101-464b-9a74-e14bb394b7fa nodeName:}" failed. No retries permitted until 2026-07-02 18:30:44.398548472 +0000 UTC m=+36.468175984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls") pod "image-registry-596cd447fb-jh626" (UID: "ed77eb69-1101-464b-9a74-e14bb394b7fa") : secret "image-registry-tls" not found
Jul 02 18:30:43.898775 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.898750    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed77eb69-1101-464b-9a74-e14bb394b7fa-trusted-ca\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.898918 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:43.898906    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls podName:684a7ad0-005b-49d6-817b-b5bc7cbfb719 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:44.398882038 +0000 UTC m=+36.468509557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls") pod "dns-default-9zq5n" (UID: "684a7ad0-005b-49d6-817b-b5bc7cbfb719") : secret "dns-default-metrics-tls" not found
Jul 02 18:30:43.899090 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.899050    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed77eb69-1101-464b-9a74-e14bb394b7fa-ca-trust-extracted\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.903275 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.903245    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed77eb69-1101-464b-9a74-e14bb394b7fa-image-registry-private-configuration\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.903389 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.903248    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed77eb69-1101-464b-9a74-e14bb394b7fa-installation-pull-secrets\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.907370 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.907336    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hbhp\" (UniqueName: \"kubernetes.io/projected/684a7ad0-005b-49d6-817b-b5bc7cbfb719-kube-api-access-4hbhp\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:43.907598 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.907574    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-bound-sa-token\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.907715 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.907682    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58cq7\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-kube-api-access-58cq7\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:43.999304 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.999266    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fa3f945-1298-44c4-8ec7-33014cddcd23-cert\") pod \"ingress-canary-qt4zw\" (UID: \"6fa3f945-1298-44c4-8ec7-33014cddcd23\") " pod="openshift-ingress-canary/ingress-canary-qt4zw"
Jul 02 18:30:43.999473 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:43.999386    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cstd6\" (UniqueName: \"kubernetes.io/projected/6fa3f945-1298-44c4-8ec7-33014cddcd23-kube-api-access-cstd6\") pod \"ingress-canary-qt4zw\" (UID: \"6fa3f945-1298-44c4-8ec7-33014cddcd23\") " pod="openshift-ingress-canary/ingress-canary-qt4zw"
Jul 02 18:30:44.001546 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.001524    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fa3f945-1298-44c4-8ec7-33014cddcd23-cert\") pod \"ingress-canary-qt4zw\" (UID: \"6fa3f945-1298-44c4-8ec7-33014cddcd23\") " pod="openshift-ingress-canary/ingress-canary-qt4zw"
Jul 02 18:30:44.008404 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.008385    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cstd6\" (UniqueName: \"kubernetes.io/projected/6fa3f945-1298-44c4-8ec7-33014cddcd23-kube-api-access-cstd6\") pod \"ingress-canary-qt4zw\" (UID: \"6fa3f945-1298-44c4-8ec7-33014cddcd23\") " pod="openshift-ingress-canary/ingress-canary-qt4zw"
Jul 02 18:30:44.078965 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.078939    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qt4zw"
Jul 02 18:30:44.165542 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.165510    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:44.210937 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.210851    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qt4zw"]
Jul 02 18:30:44.214383 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:44.214208    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa3f945_1298_44c4_8ec7_33014cddcd23.slice/crio-587784c092cc9bd80e812ddd54e9654106606da5b7ead9ffac6d2e1b2e0dab8c WatchSource:0}: Error finding container 587784c092cc9bd80e812ddd54e9654106606da5b7ead9ffac6d2e1b2e0dab8c: Status 404 returned error can't find the container with id 587784c092cc9bd80e812ddd54e9654106606da5b7ead9ffac6d2e1b2e0dab8c
Jul 02 18:30:44.340001 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.339971    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:44.396296 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.396266    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:44.402205 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.402174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:44.402288 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.402240    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:44.402325 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:44.402302    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 18:30:44.402325 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:44.402318    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596cd447fb-jh626: secret "image-registry-tls" not found
Jul 02 18:30:44.402381 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:44.402326    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 18:30:44.402381 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:44.402374    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls podName:684a7ad0-005b-49d6-817b-b5bc7cbfb719 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:45.402359869 +0000 UTC m=+37.471987374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls") pod "dns-default-9zq5n" (UID: "684a7ad0-005b-49d6-817b-b5bc7cbfb719") : secret "dns-default-metrics-tls" not found
Jul 02 18:30:44.402492 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:44.402388    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls podName:ed77eb69-1101-464b-9a74-e14bb394b7fa nodeName:}" failed. No retries permitted until 2026-07-02 18:30:45.402382393 +0000 UTC m=+37.472009898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls") pod "image-registry-596cd447fb-jh626" (UID: "ed77eb69-1101-464b-9a74-e14bb394b7fa") : secret "image-registry-tls" not found
Jul 02 18:30:44.483915 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.483840    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:30:44.484059 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.483839    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:30:44.484059 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.483840    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:30:44.486872 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.486852    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 18:30:44.486985 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.486888    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 18:30:44.487052 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.487023    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 18:30:44.487168 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.487149    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xq22w\""
Jul 02 18:30:44.488334 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.488318    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 18:30:44.488415 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.488335    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pqjw5\""
Jul 02 18:30:44.740262 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.740145    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:44.794135 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.794100    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:44.870606 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.870576    2575 generic.go:356] "Generic (PLEG): container finished" podID="98e2289a-1242-48fa-9b3e-a89b46c2c9d8" containerID="49447119f4a9c1b0e8cdebece8f51baa96e1c7dc3bc1eeb4c29df47947e90a72" exitCode=0
Jul 02 18:30:44.870777 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.870663    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6sb" event={"ID":"98e2289a-1242-48fa-9b3e-a89b46c2c9d8","Type":"ContainerDied","Data":"49447119f4a9c1b0e8cdebece8f51baa96e1c7dc3bc1eeb4c29df47947e90a72"}
Jul 02 18:30:44.871985 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:44.871963    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qt4zw" event={"ID":"6fa3f945-1298-44c4-8ec7-33014cddcd23","Type":"ContainerStarted","Data":"587784c092cc9bd80e812ddd54e9654106606da5b7ead9ffac6d2e1b2e0dab8c"}
Jul 02 18:30:45.411671 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:45.411636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:45.411671 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:45.411673    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:45.411920 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:45.411786    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 18:30:45.411920 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:45.411851    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls podName:684a7ad0-005b-49d6-817b-b5bc7cbfb719 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:47.411832804 +0000 UTC m=+39.481460313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls") pod "dns-default-9zq5n" (UID: "684a7ad0-005b-49d6-817b-b5bc7cbfb719") : secret "dns-default-metrics-tls" not found
Jul 02 18:30:45.411920 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:45.411790    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 18:30:45.411920 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:45.411880    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596cd447fb-jh626: secret "image-registry-tls" not found
Jul 02 18:30:45.411920 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:45.411925    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls podName:ed77eb69-1101-464b-9a74-e14bb394b7fa nodeName:}" failed. No retries permitted until 2026-07-02 18:30:47.411913642 +0000 UTC m=+39.481541164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls") pod "image-registry-596cd447fb-jh626" (UID: "ed77eb69-1101-464b-9a74-e14bb394b7fa") : secret "image-registry-tls" not found
Jul 02 18:30:45.878027 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:45.877993    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6sb" event={"ID":"98e2289a-1242-48fa-9b3e-a89b46c2c9d8","Type":"ContainerStarted","Data":"fbd96487d64cefc81ad1d99b66adbb83940181fd170532a8572678891944662e"}
Jul 02 18:30:45.903072 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:45.903014    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6f6sb" podStartSLOduration=6.672214047 podStartE2EDuration="36.902994573s" totalImagesPullingTime="30.230780526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:45.902377097 +0000 UTC m=+37.972004626" watchObservedRunningTime="2026-07-02 18:30:45.902994573 +0000 UTC m=+37.972622103"
Jul 02 18:30:46.028415 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:46.028374    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:46.131059 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:46.130958    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:46.881359 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:46.881328    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qt4zw" event={"ID":"6fa3f945-1298-44c4-8ec7-33014cddcd23","Type":"ContainerStarted","Data":"907c7ba2b21ec0a6fd11f67af2aa1c94f2a1923c0738f2bc71b10571bb169169"}
Jul 02 18:30:46.897924 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:46.897873    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qt4zw" podStartSLOduration=1.868990237 podStartE2EDuration="3.897857748s" totalImagesPullingTime="2.028867511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:46.897122071 +0000 UTC m=+38.966749599" watchObservedRunningTime="2026-07-02 18:30:46.897857748 +0000 UTC m=+38.967485653"
Jul 02 18:30:47.117442 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:47.117243    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:47.427456 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:47.427406    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:47.427456 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:47.427456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:47.427693 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:47.427561    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 18:30:47.427693 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:47.427582    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596cd447fb-jh626: secret "image-registry-tls" not found
Jul 02 18:30:47.427693 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:47.427625    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 18:30:47.427693 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:47.427639    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls podName:ed77eb69-1101-464b-9a74-e14bb394b7fa nodeName:}" failed. No retries permitted until 2026-07-02 18:30:51.42762367 +0000 UTC m=+43.497251180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls") pod "image-registry-596cd447fb-jh626" (UID: "ed77eb69-1101-464b-9a74-e14bb394b7fa") : secret "image-registry-tls" not found
Jul 02 18:30:47.427693 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:47.427675    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls podName:684a7ad0-005b-49d6-817b-b5bc7cbfb719 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:51.427661742 +0000 UTC m=+43.497289251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls") pod "dns-default-9zq5n" (UID: "684a7ad0-005b-49d6-817b-b5bc7cbfb719") : secret "dns-default-metrics-tls" not found
Jul 02 18:30:47.670313 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:47.670277    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:48.407536 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:48.407506    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:48.508612 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:48.508562    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:49.337268 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.337233    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs"]
Jul 02 18:30:49.353644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.353614    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs"]
Jul 02 18:30:49.353775 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.353730    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs"
Jul 02 18:30:49.356920 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.356896    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 02 18:30:49.358365 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.358345    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 02 18:30:49.358474 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.358386    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 02 18:30:49.358474 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.358390    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-rvcn2\""
Jul 02 18:30:49.358474 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.358408    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 02 18:30:49.444231 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.444177    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-8bvjs\" (UID: \"cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs"
Jul 02 18:30:49.444580 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.444242    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-8bvjs\" (UID: \"cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs"
Jul 02 18:30:49.444580 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.444266    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mvz\" (UniqueName: \"kubernetes.io/projected/cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f-kube-api-access-88mvz\") pod \"kube-storage-version-migrator-operator-846bf668f7-8bvjs\" (UID: \"cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs"
Jul 02 18:30:49.457793 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.457771    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"]
Jul 02 18:30:49.480372 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.480352    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"]
Jul 02 18:30:49.480487 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.480470    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"
Jul 02 18:30:49.483540 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.483523    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 02 18:30:49.483668 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.483638    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 02 18:30:49.483668 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.483641    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4j5qn\""
Jul 02 18:30:49.483756 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.483639    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 02 18:30:49.503511 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.503475    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn"]
Jul 02 18:30:49.503664 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.503592    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:30:49.506707 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.506683    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 02 18:30:49.506707 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.506700    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 18:30:49.506879 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.506722    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 18:30:49.506879 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.506734    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-l44tr\""
Jul 02 18:30:49.508512 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.508496    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 02 18:30:49.521682 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.521657    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"]
Jul 02 18:30:49.521788 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.521694    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn"]
Jul 02 18:30:49.521788 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.521708    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"]
Jul 02 18:30:49.521788 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.521723    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-w5w95"]
Jul 02 18:30:49.521788 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.521766    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn"
Jul 02 18:30:49.525209 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.525131    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 02 18:30:49.525339 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.525327    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 02 18:30:49.525438 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.525419    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 02 18:30:49.525535 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.525520    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-shq74\""
Jul 02 18:30:49.526039 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.526022    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 02 18:30:49.535681 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.535662    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-w5w95"]
Jul 02 18:30:49.535795 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.535780    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.539840 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.539823    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xwzbh\""
Jul 02 18:30:49.540010 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.539976    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 18:30:49.540083 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.540026    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 02 18:30:49.540083 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.539991    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 02 18:30:49.540577 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.540044    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 18:30:49.545229 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.545206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-8bvjs\" (UID: \"cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs"
Jul 02 18:30:49.545334 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.545262    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-8bvjs\" (UID: \"cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs"
Jul 02 18:30:49.545334 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.545305    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88mvz\" (UniqueName: \"kubernetes.io/projected/cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f-kube-api-access-88mvz\") pod \"kube-storage-version-migrator-operator-846bf668f7-8bvjs\" (UID: \"cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs"
Jul 02 18:30:49.546002 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.545975    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-8bvjs\" (UID: \"cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs"
Jul 02 18:30:49.548712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.548691    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pqt6r"]
Jul 02 18:30:49.549356 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.549336    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 02 18:30:49.551093 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.551076    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-8bvjs\" (UID: \"cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs"
Jul 02 18:30:49.561109 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.561084    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mvz\" (UniqueName: \"kubernetes.io/projected/cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f-kube-api-access-88mvz\") pod \"kube-storage-version-migrator-operator-846bf668f7-8bvjs\" (UID: \"cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs"
Jul 02 18:30:49.566234 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.566212    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7f6f776dd5-qrrs8"]
Jul 02 18:30:49.566332 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.566317    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pqt6r"
Jul 02 18:30:49.569184 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.569167    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rlp9j\""
Jul 02 18:30:49.569363 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.569348    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 02 18:30:49.570469 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.570455    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 02 18:30:49.588534 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.588486    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-7d9mj"]
Jul 02 18:30:49.588619 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.588607    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:49.591478 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.591462    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 02 18:30:49.591579 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.591465    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 02 18:30:49.591781 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.591768    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 02 18:30:49.591855 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.591778    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 02 18:30:49.591855 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.591787    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2ggbw\""
Jul 02 18:30:49.592012 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.591999    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 02 18:30:49.592012 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.592005    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 02 18:30:49.612755 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.612726    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pqt6r"]
Jul 02 18:30:49.612755 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.612759    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-7d9mj"]
Jul 02 18:30:49.612947 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.612781    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f6f776dd5-qrrs8"]
Jul 02 18:30:49.612947 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.612837    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:49.615979 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.615958    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 02 18:30:49.616102 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.616000    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 02 18:30:49.616159 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.616142    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 02 18:30:49.616238 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.616173    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tnk8j\""
Jul 02 18:30:49.616320 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.616306    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 02 18:30:49.623115 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.623096    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 02 18:30:49.646433 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646400    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-plgwk\" (UID: \"83478280-c42d-48e3-88d3-22001c78a49f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"
Jul 02 18:30:49.646433 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646438    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/28950053-fd3d-4df3-aae5-a38cbdda059d-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-n5jkn\" (UID: \"28950053-fd3d-4df3-aae5-a38cbdda059d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn"
Jul 02 18:30:49.646641 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646456    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5d5s\" (UniqueName: \"kubernetes.io/projected/becbcfb3-b9cc-409a-8a1c-f70ed6cb7b72-kube-api-access-r5d5s\") pod \"volume-data-source-validator-5d456fdbcd-pqt6r\" (UID: \"becbcfb3-b9cc-409a-8a1c-f70ed6cb7b72\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pqt6r"
Jul 02 18:30:49.646641 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646476    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdwlt\" (UniqueName: \"kubernetes.io/projected/28950053-fd3d-4df3-aae5-a38cbdda059d-kube-api-access-mdwlt\") pod \"service-ca-operator-5d46d9ffd4-n5jkn\" (UID: \"28950053-fd3d-4df3-aae5-a38cbdda059d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn"
Jul 02 18:30:49.646641 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646514    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhcm\" (UniqueName: \"kubernetes.io/projected/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-kube-api-access-hkhcm\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:30:49.646641 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646529    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:30:49.646641 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646585    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzkr2\" (UniqueName: \"kubernetes.io/projected/83478280-c42d-48e3-88d3-22001c78a49f-kube-api-access-fzkr2\") pod \"cluster-samples-operator-56fcf86bf-plgwk\" (UID: \"83478280-c42d-48e3-88d3-22001c78a49f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"
Jul 02 18:30:49.646641 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646609    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6846ce7e-056b-4aad-88cd-572cf4d80df3-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.646930 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646690    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6846ce7e-056b-4aad-88cd-572cf4d80df3-snapshots\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.646930 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646725    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6846ce7e-056b-4aad-88cd-572cf4d80df3-tmp\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.646930 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646757    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6846ce7e-056b-4aad-88cd-572cf4d80df3-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.646930 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646791    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/28950053-fd3d-4df3-aae5-a38cbdda059d-config\") pod \"service-ca-operator-5d46d9ffd4-n5jkn\" (UID: \"28950053-fd3d-4df3-aae5-a38cbdda059d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn"
Jul 02 18:30:49.646930 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646844    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkx9t\" (UniqueName: \"kubernetes.io/projected/6846ce7e-056b-4aad-88cd-572cf4d80df3-kube-api-access-lkx9t\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.646930 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646899    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6846ce7e-056b-4aad-88cd-572cf4d80df3-serving-cert\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.647124 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.646932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:30:49.662424 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.662406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs"
Jul 02 18:30:49.724407 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.724371    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:49.747475 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.747447    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-plgwk\" (UID: \"83478280-c42d-48e3-88d3-22001c78a49f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"
Jul 02 18:30:49.747475 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.747484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-stats-auth\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:49.747693 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:49.747604    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 18:30:49.747693 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:49.747691    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls podName:83478280-c42d-48e3-88d3-22001c78a49f nodeName:}" failed. No retries permitted until 2026-07-02 18:30:50.247664056 +0000 UTC m=+42.317291578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-plgwk" (UID: "83478280-c42d-48e3-88d3-22001c78a49f") : secret "samples-operator-tls" not found
Jul 02 18:30:49.747849 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.747603    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/28950053-fd3d-4df3-aae5-a38cbdda059d-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-n5jkn\" (UID: \"28950053-fd3d-4df3-aae5-a38cbdda059d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn"
Jul 02 18:30:49.747966 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.747896    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5d5s\" (UniqueName: \"kubernetes.io/projected/becbcfb3-b9cc-409a-8a1c-f70ed6cb7b72-kube-api-access-r5d5s\") pod \"volume-data-source-validator-5d456fdbcd-pqt6r\" (UID: \"becbcfb3-b9cc-409a-8a1c-f70ed6cb7b72\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pqt6r"
Jul 02 18:30:49.747966 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.747938    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-default-certificate\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:49.748081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.747979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4e75b84-1c82-46c8-b414-f252d7ac5633-serving-cert\") pod \"console-operator-68b757fb67-7d9mj\" (UID: \"f4e75b84-1c82-46c8-b414-f252d7ac5633\") " pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:49.748081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748053    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdwlt\" (UniqueName: \"kubernetes.io/projected/28950053-fd3d-4df3-aae5-a38cbdda059d-kube-api-access-mdwlt\") pod \"service-ca-operator-5d46d9ffd4-n5jkn\" (UID: \"28950053-fd3d-4df3-aae5-a38cbdda059d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn"
Jul 02 18:30:49.748177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhcm\" (UniqueName: \"kubernetes.io/projected/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-kube-api-access-hkhcm\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:30:49.748177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748170    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:30:49.748313 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748260    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4e75b84-1c82-46c8-b414-f252d7ac5633-trusted-ca\") pod \"console-operator-68b757fb67-7d9mj\" (UID: \"f4e75b84-1c82-46c8-b414-f252d7ac5633\") " pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:49.748431 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748412    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzkr2\" (UniqueName: \"kubernetes.io/projected/83478280-c42d-48e3-88d3-22001c78a49f-kube-api-access-fzkr2\") pod \"cluster-samples-operator-56fcf86bf-plgwk\" (UID: \"83478280-c42d-48e3-88d3-22001c78a49f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"
Jul 02 18:30:49.748487 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748447    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6846ce7e-056b-4aad-88cd-572cf4d80df3-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.748543 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748495    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2c7s\" (UniqueName: \"kubernetes.io/projected/f4e75b84-1c82-46c8-b414-f252d7ac5633-kube-api-access-r2c7s\") pod \"console-operator-68b757fb67-7d9mj\" (UID: \"f4e75b84-1c82-46c8-b414-f252d7ac5633\") " pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:49.748543 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748526    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6846ce7e-056b-4aad-88cd-572cf4d80df3-snapshots\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.748743 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748722    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6846ce7e-056b-4aad-88cd-572cf4d80df3-tmp\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.748783 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748767    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6846ce7e-056b-4aad-88cd-572cf4d80df3-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.748812 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/28950053-fd3d-4df3-aae5-a38cbdda059d-config\") pod \"service-ca-operator-5d46d9ffd4-n5jkn\" (UID: \"28950053-fd3d-4df3-aae5-a38cbdda059d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn"
Jul 02 18:30:49.748852 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh45f\" (UniqueName: \"kubernetes.io/projected/b7d790ff-691d-4fde-81c7-613493010723-kube-api-access-gh45f\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:49.748901 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748889    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkx9t\" (UniqueName: \"kubernetes.io/projected/6846ce7e-056b-4aad-88cd-572cf4d80df3-kube-api-access-lkx9t\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.748949 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748919    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:49.749003 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6846ce7e-056b-4aad-88cd-572cf4d80df3-serving-cert\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.749003 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.748996    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:49.749107 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.749030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:30:49.749107 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.749082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f4e75b84-1c82-46c8-b414-f252d7ac5633-config\") pod \"console-operator-68b757fb67-7d9mj\" (UID: \"f4e75b84-1c82-46c8-b414-f252d7ac5633\") " pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:49.749333 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.749134    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6846ce7e-056b-4aad-88cd-572cf4d80df3-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.749333 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.749242    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6846ce7e-056b-4aad-88cd-572cf4d80df3-snapshots\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.749333 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:49.749329    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 18:30:49.749491 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.749034    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:30:49.749491 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.749443    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6846ce7e-056b-4aad-88cd-572cf4d80df3-tmp\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.749590 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:49.749513    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls podName:5b2c35b5-1bcf-4674-9b53-60abc6db4c07 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:50.249391015 +0000 UTC m=+42.319018540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gstlk" (UID: "5b2c35b5-1bcf-4674-9b53-60abc6db4c07") : secret "cluster-monitoring-operator-tls" not found
Jul 02 18:30:49.749705 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.749688    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/28950053-fd3d-4df3-aae5-a38cbdda059d-config\") pod \"service-ca-operator-5d46d9ffd4-n5jkn\" (UID: \"28950053-fd3d-4df3-aae5-a38cbdda059d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn"
Jul 02 18:30:49.750206 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.750178    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/28950053-fd3d-4df3-aae5-a38cbdda059d-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-n5jkn\" (UID: \"28950053-fd3d-4df3-aae5-a38cbdda059d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn"
Jul 02 18:30:49.750492 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.750470    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6846ce7e-056b-4aad-88cd-572cf4d80df3-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.751608 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.751581    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6846ce7e-056b-4aad-88cd-572cf4d80df3-serving-cert\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.759628 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.759448    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhcm\" (UniqueName: \"kubernetes.io/projected/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-kube-api-access-hkhcm\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:30:49.761359 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.761336    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzkr2\" (UniqueName: \"kubernetes.io/projected/83478280-c42d-48e3-88d3-22001c78a49f-kube-api-access-fzkr2\") pod \"cluster-samples-operator-56fcf86bf-plgwk\" (UID: \"83478280-c42d-48e3-88d3-22001c78a49f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"
Jul 02 18:30:49.761922 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.761903    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkx9t\" (UniqueName: \"kubernetes.io/projected/6846ce7e-056b-4aad-88cd-572cf4d80df3-kube-api-access-lkx9t\") pod \"insights-operator-5c847dcfc4-w5w95\" (UID: \"6846ce7e-056b-4aad-88cd-572cf4d80df3\") " pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.762152 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.762135    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdwlt\" (UniqueName: \"kubernetes.io/projected/28950053-fd3d-4df3-aae5-a38cbdda059d-kube-api-access-mdwlt\") pod \"service-ca-operator-5d46d9ffd4-n5jkn\" (UID: \"28950053-fd3d-4df3-aae5-a38cbdda059d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn"
Jul 02 18:30:49.762271 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.762249    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5d5s\" (UniqueName: \"kubernetes.io/projected/becbcfb3-b9cc-409a-8a1c-f70ed6cb7b72-kube-api-access-r5d5s\") pod \"volume-data-source-validator-5d456fdbcd-pqt6r\" (UID: \"becbcfb3-b9cc-409a-8a1c-f70ed6cb7b72\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pqt6r"
Jul 02 18:30:49.807620 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.807592    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs"]
Jul 02 18:30:49.809590 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:49.809563    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb708ea_5ecd_4a2c_a4d4_08565fe50f2f.slice/crio-0e80c9ebca840509f1aee10bc994a1b814650f04edcd8fd61a2218984683a510 WatchSource:0}: Error finding container 0e80c9ebca840509f1aee10bc994a1b814650f04edcd8fd61a2218984683a510: Status 404 returned error can't find the container with id 0e80c9ebca840509f1aee10bc994a1b814650f04edcd8fd61a2218984683a510
Jul 02 18:30:49.809914 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.809874    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:49.830595 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.830560    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn"
Jul 02 18:30:49.844722 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.844669    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-w5w95"
Jul 02 18:30:49.849999 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.849977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh45f\" (UniqueName: \"kubernetes.io/projected/b7d790ff-691d-4fde-81c7-613493010723-kube-api-access-gh45f\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:49.850118 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.850027    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:49.850118 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.850076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:49.850251 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.850128    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f4e75b84-1c82-46c8-b414-f252d7ac5633-config\") pod \"console-operator-68b757fb67-7d9mj\" (UID: \"f4e75b84-1c82-46c8-b414-f252d7ac5633\") " pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:49.850251 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.850166    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-stats-auth\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:49.850251 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:49.850221    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 18:30:49.850251 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:49.850231    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle podName:b7d790ff-691d-4fde-81c7-613493010723 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:50.350212217 +0000 UTC m=+42.419839724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle") pod "router-default-7f6f776dd5-qrrs8" (UID: "b7d790ff-691d-4fde-81c7-613493010723") : configmap references non-existent config key: service-ca.crt
Jul 02 18:30:49.850444 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.850257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-default-certificate\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:49.850444 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:49.850267    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs podName:b7d790ff-691d-4fde-81c7-613493010723 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:50.350254818 +0000 UTC m=+42.419882324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs") pod "router-default-7f6f776dd5-qrrs8" (UID: "b7d790ff-691d-4fde-81c7-613493010723") : secret "router-metrics-certs-default" not found
Jul 02 18:30:49.850444 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.850299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4e75b84-1c82-46c8-b414-f252d7ac5633-serving-cert\") pod \"console-operator-68b757fb67-7d9mj\" (UID: \"f4e75b84-1c82-46c8-b414-f252d7ac5633\") " pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:49.850444 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.850362    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4e75b84-1c82-46c8-b414-f252d7ac5633-trusted-ca\") pod \"console-operator-68b757fb67-7d9mj\" (UID: \"f4e75b84-1c82-46c8-b414-f252d7ac5633\") " pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:49.850444 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.850392    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2c7s\" (UniqueName: \"kubernetes.io/projected/f4e75b84-1c82-46c8-b414-f252d7ac5633-kube-api-access-r2c7s\") pod \"console-operator-68b757fb67-7d9mj\" (UID: \"f4e75b84-1c82-46c8-b414-f252d7ac5633\") " pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:49.855022 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.851598    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f4e75b84-1c82-46c8-b414-f252d7ac5633-config\") pod \"console-operator-68b757fb67-7d9mj\" (UID: \"f4e75b84-1c82-46c8-b414-f252d7ac5633\") " pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:49.855022 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.851688    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4e75b84-1c82-46c8-b414-f252d7ac5633-trusted-ca\") pod \"console-operator-68b757fb67-7d9mj\" (UID: \"f4e75b84-1c82-46c8-b414-f252d7ac5633\") " pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:49.855948 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.855922    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-stats-auth\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:49.856040 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.856023    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-default-certificate\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:49.856919 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.856900    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4e75b84-1c82-46c8-b414-f252d7ac5633-serving-cert\") pod \"console-operator-68b757fb67-7d9mj\" (UID: \"f4e75b84-1c82-46c8-b414-f252d7ac5633\") " pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:49.858385 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.858365    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh45f\" (UniqueName: \"kubernetes.io/projected/b7d790ff-691d-4fde-81c7-613493010723-kube-api-access-gh45f\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:49.858928 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.858908    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2c7s\" (UniqueName: \"kubernetes.io/projected/f4e75b84-1c82-46c8-b414-f252d7ac5633-kube-api-access-r2c7s\") pod \"console-operator-68b757fb67-7d9mj\" (UID: \"f4e75b84-1c82-46c8-b414-f252d7ac5633\") " pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:49.874602 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.874584    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pqt6r"
Jul 02 18:30:49.889897 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.889868    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs" event={"ID":"cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f","Type":"ContainerStarted","Data":"0e80c9ebca840509f1aee10bc994a1b814650f04edcd8fd61a2218984683a510"}
Jul 02 18:30:49.923545 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:49.922472    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:50.019929 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:50.019861    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn"]
Jul 02 18:30:50.024851 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:50.024819    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28950053_fd3d_4df3_aae5_a38cbdda059d.slice/crio-818cb689d3373c505b018136d450bfa1781102af85c363c69871f8ea2eea84bd WatchSource:0}: Error finding container 818cb689d3373c505b018136d450bfa1781102af85c363c69871f8ea2eea84bd: Status 404 returned error can't find the container with id 818cb689d3373c505b018136d450bfa1781102af85c363c69871f8ea2eea84bd
Jul 02 18:30:50.051171 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:50.051145    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-w5w95"]
Jul 02 18:30:50.053775 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:50.053750    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6846ce7e_056b_4aad_88cd_572cf4d80df3.slice/crio-daee299b707e244c1babcf81521c57b7edac5b5ca4e8dece7d4cf4e54e6a939d WatchSource:0}: Error finding container daee299b707e244c1babcf81521c57b7edac5b5ca4e8dece7d4cf4e54e6a939d: Status 404 returned error can't find the container with id daee299b707e244c1babcf81521c57b7edac5b5ca4e8dece7d4cf4e54e6a939d
Jul 02 18:30:50.077277 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:50.077211    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pqt6r"]
Jul 02 18:30:50.078888 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:50.078866    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecbcfb3_b9cc_409a_8a1c_f70ed6cb7b72.slice/crio-e012b03070e34795ad814088cbdfb18ce2ffbb31ca203399f896380b70d0ca48 WatchSource:0}: Error finding container e012b03070e34795ad814088cbdfb18ce2ffbb31ca203399f896380b70d0ca48: Status 404 returned error can't find the container with id e012b03070e34795ad814088cbdfb18ce2ffbb31ca203399f896380b70d0ca48
Jul 02 18:30:50.102017 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:50.101946    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-7d9mj"]
Jul 02 18:30:50.104800 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:30:50.104774    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e75b84_1c82_46c8_b414_f252d7ac5633.slice/crio-d7e910141d22d11ad171d76a53e3514854b61015a105644bd3c22eff7e057942 WatchSource:0}: Error finding container d7e910141d22d11ad171d76a53e3514854b61015a105644bd3c22eff7e057942: Status 404 returned error can't find the container with id d7e910141d22d11ad171d76a53e3514854b61015a105644bd3c22eff7e057942
Jul 02 18:30:50.254208 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:50.254161    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:30:50.254357 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:50.254220    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-plgwk\" (UID: \"83478280-c42d-48e3-88d3-22001c78a49f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"
Jul 02 18:30:50.254357 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:50.254311    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 18:30:50.254425 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:50.254367    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 18:30:50.254425 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:50.254387    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls podName:5b2c35b5-1bcf-4674-9b53-60abc6db4c07 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:51.254368024 +0000 UTC m=+43.323995549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gstlk" (UID: "5b2c35b5-1bcf-4674-9b53-60abc6db4c07") : secret "cluster-monitoring-operator-tls" not found
Jul 02 18:30:50.254425 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:50.254412    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls podName:83478280-c42d-48e3-88d3-22001c78a49f nodeName:}" failed. No retries permitted until 2026-07-02 18:30:51.254399789 +0000 UTC m=+43.324027296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-plgwk" (UID: "83478280-c42d-48e3-88d3-22001c78a49f") : secret "samples-operator-tls" not found
Jul 02 18:30:50.355309 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:50.355229    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:50.355309 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:50.355282    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:50.355473 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:50.355410    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle podName:b7d790ff-691d-4fde-81c7-613493010723 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:51.355391989 +0000 UTC m=+43.425019513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle") pod "router-default-7f6f776dd5-qrrs8" (UID: "b7d790ff-691d-4fde-81c7-613493010723") : configmap references non-existent config key: service-ca.crt
Jul 02 18:30:50.355473 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:50.355414    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 18:30:50.355473 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:50.355451    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs podName:b7d790ff-691d-4fde-81c7-613493010723 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:51.355442193 +0000 UTC m=+43.425069699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs") pod "router-default-7f6f776dd5-qrrs8" (UID: "b7d790ff-691d-4fde-81c7-613493010723") : secret "router-metrics-certs-default" not found
Jul 02 18:30:50.894816 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:50.894767    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pqt6r" event={"ID":"becbcfb3-b9cc-409a-8a1c-f70ed6cb7b72","Type":"ContainerStarted","Data":"e012b03070e34795ad814088cbdfb18ce2ffbb31ca203399f896380b70d0ca48"}
Jul 02 18:30:50.897059 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:50.896982    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-7d9mj" event={"ID":"f4e75b84-1c82-46c8-b414-f252d7ac5633","Type":"ContainerStarted","Data":"d7e910141d22d11ad171d76a53e3514854b61015a105644bd3c22eff7e057942"}
Jul 02 18:30:50.898298 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:50.898268    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-w5w95" event={"ID":"6846ce7e-056b-4aad-88cd-572cf4d80df3","Type":"ContainerStarted","Data":"daee299b707e244c1babcf81521c57b7edac5b5ca4e8dece7d4cf4e54e6a939d"}
Jul 02 18:30:50.899896 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:50.899848    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn" event={"ID":"28950053-fd3d-4df3-aae5-a38cbdda059d","Type":"ContainerStarted","Data":"818cb689d3373c505b018136d450bfa1781102af85c363c69871f8ea2eea84bd"}
Jul 02 18:30:51.264374 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:51.264129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-plgwk\" (UID: \"83478280-c42d-48e3-88d3-22001c78a49f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"
Jul 02 18:30:51.264374 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:51.264331    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 18:30:51.264588 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:51.264408    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls podName:83478280-c42d-48e3-88d3-22001c78a49f nodeName:}" failed. No retries permitted until 2026-07-02 18:30:53.264386226 +0000 UTC m=+45.334013735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-plgwk" (UID: "83478280-c42d-48e3-88d3-22001c78a49f") : secret "samples-operator-tls" not found
Jul 02 18:30:51.264825 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:51.264750    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:30:51.264959 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:51.264891    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 18:30:51.264959 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:51.264946    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls podName:5b2c35b5-1bcf-4674-9b53-60abc6db4c07 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:53.264931218 +0000 UTC m=+45.334558730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gstlk" (UID: "5b2c35b5-1bcf-4674-9b53-60abc6db4c07") : secret "cluster-monitoring-operator-tls" not found
Jul 02 18:30:51.365643 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:51.365562    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:51.365816 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:51.365752    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle podName:b7d790ff-691d-4fde-81c7-613493010723 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:53.365728337 +0000 UTC m=+45.435355845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle") pod "router-default-7f6f776dd5-qrrs8" (UID: "b7d790ff-691d-4fde-81c7-613493010723") : configmap references non-existent config key: service-ca.crt
Jul 02 18:30:51.365957 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:51.365931    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:51.366104 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:51.366084    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 18:30:51.366165 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:51.366153    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs podName:b7d790ff-691d-4fde-81c7-613493010723 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:53.366135589 +0000 UTC m=+45.435763098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs") pod "router-default-7f6f776dd5-qrrs8" (UID: "b7d790ff-691d-4fde-81c7-613493010723") : secret "router-metrics-certs-default" not found
Jul 02 18:30:51.467335 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:51.467165    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:51.467335 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:51.467231    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:51.467545 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:51.467495    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 18:30:51.467545 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:51.467514    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596cd447fb-jh626: secret "image-registry-tls" not found
Jul 02 18:30:51.467655 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:51.467572    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls podName:ed77eb69-1101-464b-9a74-e14bb394b7fa nodeName:}" failed. No retries permitted until 2026-07-02 18:30:59.467554016 +0000 UTC m=+51.537181536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls") pod "image-registry-596cd447fb-jh626" (UID: "ed77eb69-1101-464b-9a74-e14bb394b7fa") : secret "image-registry-tls" not found
Jul 02 18:30:51.467860 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:51.467787    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 18:30:51.467860 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:51.467839    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls podName:684a7ad0-005b-49d6-817b-b5bc7cbfb719 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:59.467824808 +0000 UTC m=+51.537452329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls") pod "dns-default-9zq5n" (UID: "684a7ad0-005b-49d6-817b-b5bc7cbfb719") : secret "dns-default-metrics-tls" not found
Jul 02 18:30:53.282907 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:53.282872    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:30:53.282907 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:53.282922    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-plgwk\" (UID: \"83478280-c42d-48e3-88d3-22001c78a49f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"
Jul 02 18:30:53.283452 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:53.283094    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 18:30:53.283452 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:53.283175    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls podName:83478280-c42d-48e3-88d3-22001c78a49f nodeName:}" failed. No retries permitted until 2026-07-02 18:30:57.283159345 +0000 UTC m=+49.352786856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-plgwk" (UID: "83478280-c42d-48e3-88d3-22001c78a49f") : secret "samples-operator-tls" not found
Jul 02 18:30:53.283452 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:53.283102    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 18:30:53.283452 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:53.283249    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls podName:5b2c35b5-1bcf-4674-9b53-60abc6db4c07 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:57.283236197 +0000 UTC m=+49.352863707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gstlk" (UID: "5b2c35b5-1bcf-4674-9b53-60abc6db4c07") : secret "cluster-monitoring-operator-tls" not found
Jul 02 18:30:53.384362 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:53.384326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:53.384527 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:53.384390    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:53.384583 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:53.384527    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle podName:b7d790ff-691d-4fde-81c7-613493010723 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:57.384507223 +0000 UTC m=+49.454134753 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle") pod "router-default-7f6f776dd5-qrrs8" (UID: "b7d790ff-691d-4fde-81c7-613493010723") : configmap references non-existent config key: service-ca.crt
Jul 02 18:30:53.384583 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:53.384555    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 18:30:53.384685 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:53.384619    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs podName:b7d790ff-691d-4fde-81c7-613493010723 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:57.384601793 +0000 UTC m=+49.454229300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs") pod "router-default-7f6f776dd5-qrrs8" (UID: "b7d790ff-691d-4fde-81c7-613493010723") : secret "router-metrics-certs-default" not found
Jul 02 18:30:54.915496 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:54.915453    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn" event={"ID":"28950053-fd3d-4df3-aae5-a38cbdda059d","Type":"ContainerStarted","Data":"04e8861f3a2335f6c2b7a36f8593d8a28c61b456d8b5f2eb1f1fcdcede86ee46"}
Jul 02 18:30:54.917306 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:54.917275    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pqt6r" event={"ID":"becbcfb3-b9cc-409a-8a1c-f70ed6cb7b72","Type":"ContainerStarted","Data":"b7ce25440ac4452100d0a9b1c42d1782627ed2fab769a865b7d69b76991eefe2"}
Jul 02 18:30:54.933984 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:54.933884    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn" podStartSLOduration=1.215648145 podStartE2EDuration="5.933859922s" totalImagesPullingTime="4.718211777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:54.932831873 +0000 UTC m=+47.002459426" watchObservedRunningTime="2026-07-02 18:30:54.933859922 +0000 UTC m=+47.003487453"
Jul 02 18:30:54.964297 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:54.963766    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pqt6r" podStartSLOduration=1.3887835960000001 podStartE2EDuration="5.963745327s" totalImagesPullingTime="4.574961731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:54.962324474 +0000 UTC m=+47.031952015" watchObservedRunningTime="2026-07-02 18:30:54.963745327 +0000 UTC m=+47.033372856"
Jul 02 18:30:55.054398 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.054361    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:55.110910 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.110876    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:55.728075 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.728040    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dp4vg"]
Jul 02 18:30:55.753151 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.753127    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dp4vg"]
Jul 02 18:30:55.753316 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.753244    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:55.756088 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.756054    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 18:30:55.756088 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.756058    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 18:30:55.756300 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.756164    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fqsrq\""
Jul 02 18:30:55.776075 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.776052    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:55.899292 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.899260    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:55.907537 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.907504    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:55.907691 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.907548    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/572ae32e-0565-40c9-97eb-21161b4fb2c9-data-volume\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:55.907691 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.907629    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2g2h\" (UniqueName: \"kubernetes.io/projected/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-api-access-s2g2h\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:55.907803 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.907725    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:55.907855 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.907840    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/572ae32e-0565-40c9-97eb-21161b4fb2c9-crio-socket\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:55.922612 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.922581    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs" event={"ID":"cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f","Type":"ContainerStarted","Data":"07f6fcd2563d39936b5b8b90bf3cc2ff3e948ece52a584c7c35b8b7f29091d4b"}
Jul 02 18:30:55.924340 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.924320    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/0.log"
Jul 02 18:30:55.924472 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.924355    2575 generic.go:356] "Generic (PLEG): container finished" podID="f4e75b84-1c82-46c8-b414-f252d7ac5633" containerID="2664584a7d64393e0e0c38936ddb6d17f6e094b7f4d3b1bb40c234e6e11609bd" exitCode=255
Jul 02 18:30:55.924472 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.924416    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-7d9mj" event={"ID":"f4e75b84-1c82-46c8-b414-f252d7ac5633","Type":"ContainerDied","Data":"2664584a7d64393e0e0c38936ddb6d17f6e094b7f4d3b1bb40c234e6e11609bd"}
Jul 02 18:30:55.924684 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.924667    2575 scope.go:117] "RemoveContainer" containerID="2664584a7d64393e0e0c38936ddb6d17f6e094b7f4d3b1bb40c234e6e11609bd"
Jul 02 18:30:55.926089 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.926051    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-w5w95" event={"ID":"6846ce7e-056b-4aad-88cd-572cf4d80df3","Type":"ContainerStarted","Data":"df207a9ca03715d2c08d500bd5bc5ff82ba8320fd83ac9ecc44cfe53b797af1e"}
Jul 02 18:30:55.940110 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.940072    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs" podStartSLOduration=2.004978037 podStartE2EDuration="6.94005986s" totalImagesPullingTime="4.935081823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:55.939136193 +0000 UTC m=+48.008763722" watchObservedRunningTime="2026-07-02 18:30:55.94005986 +0000 UTC m=+48.009687388"
Jul 02 18:30:55.972984 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.972667    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-w5w95" podStartSLOduration=2.282337208 podStartE2EDuration="6.9726477s" totalImagesPullingTime="4.690310492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:55.972113282 +0000 UTC m=+48.041740815" watchObservedRunningTime="2026-07-02 18:30:55.9726477 +0000 UTC m=+48.042275234"
Jul 02 18:30:55.976781 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:55.976689    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:56.008897 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.008857    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:56.009086 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.008954    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/572ae32e-0565-40c9-97eb-21161b4fb2c9-crio-socket\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:56.009086 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:56.009000    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 18:30:56.009086 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.009011    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:56.009086 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.009042    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/572ae32e-0565-40c9-97eb-21161b4fb2c9-data-volume\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:56.009086 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:56.009071    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-rbac-proxy-cm podName:572ae32e-0565-40c9-97eb-21161b4fb2c9 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:56.509048843 +0000 UTC m=+48.578676363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dp4vg" (UID: "572ae32e-0565-40c9-97eb-21161b4fb2c9") : configmap "kube-rbac-proxy" not found
Jul 02 18:30:56.009390 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.009163    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2g2h\" (UniqueName: \"kubernetes.io/projected/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-api-access-s2g2h\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:56.009390 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.009219    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/572ae32e-0565-40c9-97eb-21161b4fb2c9-crio-socket\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:56.009390 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:56.009322    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 18:30:56.009390 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.009354    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/572ae32e-0565-40c9-97eb-21161b4fb2c9-data-volume\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:56.009390 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:56.009371    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls podName:572ae32e-0565-40c9-97eb-21161b4fb2c9 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:56.509355226 +0000 UTC m=+48.578982797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dp4vg" (UID: "572ae32e-0565-40c9-97eb-21161b4fb2c9") : secret "insights-runtime-extractor-tls" not found
Jul 02 18:30:56.018680 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.018644    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2g2h\" (UniqueName: \"kubernetes.io/projected/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-api-access-s2g2h\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:56.318420 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.318385    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:56.419408 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.419371    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:56.514817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.514778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:56.515064 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.514859    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:56.515064 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:56.514951    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 18:30:56.515064 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:56.514955    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 18:30:56.515064 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:56.515022    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-rbac-proxy-cm podName:572ae32e-0565-40c9-97eb-21161b4fb2c9 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:57.51500697 +0000 UTC m=+49.584634477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dp4vg" (UID: "572ae32e-0565-40c9-97eb-21161b4fb2c9") : configmap "kube-rbac-proxy" not found
Jul 02 18:30:56.515064 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:56.515037    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls podName:572ae32e-0565-40c9-97eb-21161b4fb2c9 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:57.515030638 +0000 UTC m=+49.584658144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dp4vg" (UID: "572ae32e-0565-40c9-97eb-21161b4fb2c9") : secret "insights-runtime-extractor-tls" not found
Jul 02 18:30:56.519102 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.519064    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:56.622573 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.622480    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:56.806469 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.806431    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:56.907271 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.907137    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:56.930504 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.930479    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 18:30:56.930920 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.930870    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/0.log"
Jul 02 18:30:56.930920 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.930903    2575 generic.go:356] "Generic (PLEG): container finished" podID="f4e75b84-1c82-46c8-b414-f252d7ac5633" containerID="f181537730819b6d8e822c871712088ccb345541bb5a635421a225c3c44c1683" exitCode=255
Jul 02 18:30:56.931017 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.930934    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-7d9mj" event={"ID":"f4e75b84-1c82-46c8-b414-f252d7ac5633","Type":"ContainerDied","Data":"f181537730819b6d8e822c871712088ccb345541bb5a635421a225c3c44c1683"}
Jul 02 18:30:56.931017 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.930994    2575 scope.go:117] "RemoveContainer" containerID="2664584a7d64393e0e0c38936ddb6d17f6e094b7f4d3b1bb40c234e6e11609bd"
Jul 02 18:30:56.931377 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:56.931348    2575 scope.go:117] "RemoveContainer" containerID="f181537730819b6d8e822c871712088ccb345541bb5a635421a225c3c44c1683"
Jul 02 18:30:56.931522 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:56.931503    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-7d9mj_openshift-console-operator(f4e75b84-1c82-46c8-b414-f252d7ac5633)\"" pod="openshift-console-operator/console-operator-68b757fb67-7d9mj" podUID="f4e75b84-1c82-46c8-b414-f252d7ac5633"
Jul 02 18:30:57.077697 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:57.077659    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:57.185481 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:57.185393    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:57.305942 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:57.305916    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6mtkp_d5a96438-0081-4f62-aa03-cd7cf1bb0f2c/dns-node-resolver/0.log"
Jul 02 18:30:57.323422 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:57.323387    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-plgwk\" (UID: \"83478280-c42d-48e3-88d3-22001c78a49f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"
Jul 02 18:30:57.323585 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:57.323547    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 18:30:57.323621 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:57.323576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:30:57.323621 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:57.323614    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls podName:83478280-c42d-48e3-88d3-22001c78a49f nodeName:}" failed. No retries permitted until 2026-07-02 18:31:05.323594265 +0000 UTC m=+57.393221794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-plgwk" (UID: "83478280-c42d-48e3-88d3-22001c78a49f") : secret "samples-operator-tls" not found
Jul 02 18:30:57.323689 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:57.323650    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 18:30:57.323718 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:57.323698    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls podName:5b2c35b5-1bcf-4674-9b53-60abc6db4c07 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:05.323686798 +0000 UTC m=+57.393314321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gstlk" (UID: "5b2c35b5-1bcf-4674-9b53-60abc6db4c07") : secret "cluster-monitoring-operator-tls" not found
Jul 02 18:30:57.424055 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:57.424014    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:57.424284 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:57.424085    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:30:57.424284 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:57.424210    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle podName:b7d790ff-691d-4fde-81c7-613493010723 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:05.42417285 +0000 UTC m=+57.493800359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle") pod "router-default-7f6f776dd5-qrrs8" (UID: "b7d790ff-691d-4fde-81c7-613493010723") : configmap references non-existent config key: service-ca.crt
Jul 02 18:30:57.424284 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:57.424264    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 18:30:57.424454 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:57.424324    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs podName:b7d790ff-691d-4fde-81c7-613493010723 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:05.424312372 +0000 UTC m=+57.493939883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs") pod "router-default-7f6f776dd5-qrrs8" (UID: "b7d790ff-691d-4fde-81c7-613493010723") : secret "router-metrics-certs-default" not found
Jul 02 18:30:57.525321 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:57.525239    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:57.525474 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:57.525349    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:57.525474 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:57.525385    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 18:30:57.525474 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:57.525452    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls podName:572ae32e-0565-40c9-97eb-21161b4fb2c9 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:59.525436871 +0000 UTC m=+51.595064377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dp4vg" (UID: "572ae32e-0565-40c9-97eb-21161b4fb2c9") : secret "insights-runtime-extractor-tls" not found
Jul 02 18:30:57.525474 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:57.525456    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 18:30:57.525627 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:57.525504    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-rbac-proxy-cm podName:572ae32e-0565-40c9-97eb-21161b4fb2c9 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:59.525491145 +0000 UTC m=+51.595118655 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dp4vg" (UID: "572ae32e-0565-40c9-97eb-21161b4fb2c9") : configmap "kube-rbac-proxy" not found
Jul 02 18:30:57.545376 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:57.545346    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:57.600138 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:57.600101    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:57.940329 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:57.940294    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 18:30:57.940775 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:57.940608    2575 scope.go:117] "RemoveContainer" containerID="f181537730819b6d8e822c871712088ccb345541bb5a635421a225c3c44c1683"
Jul 02 18:30:57.940775 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:57.940751    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-7d9mj_openshift-console-operator(f4e75b84-1c82-46c8-b414-f252d7ac5633)\"" pod="openshift-console-operator/console-operator-68b757fb67-7d9mj" podUID="f4e75b84-1c82-46c8-b414-f252d7ac5633"
Jul 02 18:30:58.106962 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:58.106931    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6cwjc_02cf5868-e870-47eb-870f-c87dcabdae5a/node-ca/0.log"
Jul 02 18:30:58.746717 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:58.746676    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:58.835561 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:58.835523    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:59.107321 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:59.107292    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qt4zw_6fa3f945-1298-44c4-8ec7-33014cddcd23/serve-healthcheck-canary/0.log"
Jul 02 18:30:59.538230 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:59.538130    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:59.543136 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:59.543115    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:59.543251 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:59.543231    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:59.543289 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:59.543278    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 18:30:59.543289 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:59.543282    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:30:59.543351 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:59.543334    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls podName:572ae32e-0565-40c9-97eb-21161b4fb2c9 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:03.543318124 +0000 UTC m=+55.612945646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dp4vg" (UID: "572ae32e-0565-40c9-97eb-21161b4fb2c9") : secret "insights-runtime-extractor-tls" not found
Jul 02 18:30:59.543393 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:59.543352    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 18:30:59.543393 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:59.543364    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596cd447fb-jh626: secret "image-registry-tls" not found
Jul 02 18:30:59.543460 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:59.543406    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls podName:ed77eb69-1101-464b-9a74-e14bb394b7fa nodeName:}" failed. No retries permitted until 2026-07-02 18:31:15.543394532 +0000 UTC m=+67.613022051 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls") pod "image-registry-596cd447fb-jh626" (UID: "ed77eb69-1101-464b-9a74-e14bb394b7fa") : secret "image-registry-tls" not found
Jul 02 18:30:59.543460 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:59.543408    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:30:59.543540 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:59.543465    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 18:30:59.543540 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:59.543499    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls podName:684a7ad0-005b-49d6-817b-b5bc7cbfb719 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:15.543490019 +0000 UTC m=+67.613117544 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls") pod "dns-default-9zq5n" (UID: "684a7ad0-005b-49d6-817b-b5bc7cbfb719") : secret "dns-default-metrics-tls" not found
Jul 02 18:30:59.555889 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:59.555869    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/572ae32e-0565-40c9-97eb-21161b4fb2c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:30:59.637920 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:59.637885    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:30:59.923538 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:59.923496    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:59.923744 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:59.923563    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:30:59.923948 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:30:59.923933    2575 scope.go:117] "RemoveContainer" containerID="f181537730819b6d8e822c871712088ccb345541bb5a635421a225c3c44c1683"
Jul 02 18:30:59.924125 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:30:59.924102    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-7d9mj_openshift-console-operator(f4e75b84-1c82-46c8-b414-f252d7ac5633)\"" pod="openshift-console-operator/console-operator-68b757fb67-7d9mj" podUID="f4e75b84-1c82-46c8-b414-f252d7ac5633"
Jul 02 18:31:02.954259 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:02.954230    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:03.122982 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:03.122943    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:03.222714 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:03.222625    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:03.321963 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:03.321915    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:03.422097 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:03.422061    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:03.488889 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:03.488805    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:03.577007 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:03.576974    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:31:03.577214 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:31:03.577145    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 18:31:03.577289 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:31:03.577245    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls podName:572ae32e-0565-40c9-97eb-21161b4fb2c9 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:11.577222268 +0000 UTC m=+63.646849791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dp4vg" (UID: "572ae32e-0565-40c9-97eb-21161b4fb2c9") : secret "insights-runtime-extractor-tls" not found
Jul 02 18:31:04.059395 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:04.059360    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:04.478969 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:04.478885    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:04.851615 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:04.851581    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:05.048633 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:05.048601    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:05.391633 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:05.391596    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-plgwk\" (UID: \"83478280-c42d-48e3-88d3-22001c78a49f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"
Jul 02 18:31:05.392188 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:05.391732    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:31:05.392188 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:31:05.391820    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 18:31:05.392188 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:31:05.391884    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls podName:5b2c35b5-1bcf-4674-9b53-60abc6db4c07 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:21.39187009 +0000 UTC m=+73.461497595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gstlk" (UID: "5b2c35b5-1bcf-4674-9b53-60abc6db4c07") : secret "cluster-monitoring-operator-tls" not found
Jul 02 18:31:05.394306 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:05.394283    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83478280-c42d-48e3-88d3-22001c78a49f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-plgwk\" (UID: \"83478280-c42d-48e3-88d3-22001c78a49f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"
Jul 02 18:31:05.492898 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:05.492852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:31:05.493074 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:05.492939    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:31:05.493117 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:31:05.493093    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle podName:b7d790ff-691d-4fde-81c7-613493010723 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:21.49307413 +0000 UTC m=+73.562701641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle") pod "router-default-7f6f776dd5-qrrs8" (UID: "b7d790ff-691d-4fde-81c7-613493010723") : configmap references non-existent config key: service-ca.crt
Jul 02 18:31:05.493117 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:31:05.493109    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 18:31:05.493240 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:31:05.493167    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs podName:b7d790ff-691d-4fde-81c7-613493010723 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:21.493150842 +0000 UTC m=+73.562778347 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs") pod "router-default-7f6f776dd5-qrrs8" (UID: "b7d790ff-691d-4fde-81c7-613493010723") : secret "router-metrics-certs-default" not found
Jul 02 18:31:05.645366 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:05.645282    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:05.689870 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:05.689843    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"
Jul 02 18:31:05.841641 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:05.841614    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk"]
Jul 02 18:31:05.963795 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:05.963763    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk" event={"ID":"83478280-c42d-48e3-88d3-22001c78a49f","Type":"ContainerStarted","Data":"a65724406a002f66002aad0828d4841f29a1e7989acb23dc75917c45a1793292"}
Jul 02 18:31:06.068171 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:06.068132    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:06.277657 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:06.277572    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:06.457744 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:06.457697    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:06.760953 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:06.760912    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:06.861255 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:06.861218    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:07.249987 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:07.249942    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:07.456417 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:07.456385    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:07.556044 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:07.556013    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:07.847569 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:07.847490    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:07.856933 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:07.856910    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkdd4"
Jul 02 18:31:08.448464 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:08.448373    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:08.952456 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:08.952428    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:08.974811 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:08.974776    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk" event={"ID":"83478280-c42d-48e3-88d3-22001c78a49f","Type":"ContainerStarted","Data":"46689f4a3291bb149f8b8267cf8ed1619f9647e7eea828e6fc4da70378336999"}
Jul 02 18:31:08.974953 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:08.974843    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk" event={"ID":"83478280-c42d-48e3-88d3-22001c78a49f","Type":"ContainerStarted","Data":"b1da30da09daf15b4e9d465f9dd63c65734bc77954657162e66af4cedfada684"}
Jul 02 18:31:08.991242 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:08.991183    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-plgwk" podStartSLOduration=17.913584196 podStartE2EDuration="19.991170642s" totalImagesPullingTime="2.077586446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:08.990721077 +0000 UTC m=+61.060348604" watchObservedRunningTime="2026-07-02 18:31:08.991170642 +0000 UTC m=+61.060798172"
Jul 02 18:31:09.052734 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:09.052697    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:09.242425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:09.242340    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:09.446106 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:09.446069    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:09.644665 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:09.644630    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:10.452445 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:10.452411    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:10.819614 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:10.819582    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:11.041830 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.041798    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:11.241731 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.241648    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:11.445176 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.445137    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:11.483357 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.483328    2575 scope.go:117] "RemoveContainer" containerID="f181537730819b6d8e822c871712088ccb345541bb5a635421a225c3c44c1683"
Jul 02 18:31:11.646105 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.646066    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:31:11.648435 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.648408    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/572ae32e-0565-40c9-97eb-21161b4fb2c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dp4vg\" (UID: \"572ae32e-0565-40c9-97eb-21161b4fb2c9\") " pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:31:11.664932 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.664909    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fqsrq\""
Jul 02 18:31:11.673255 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.673236    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dp4vg"
Jul 02 18:31:11.798906 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.798876    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dp4vg"]
Jul 02 18:31:11.805261 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:31:11.804688    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572ae32e_0565_40c9_97eb_21161b4fb2c9.slice/crio-133c0a2af4f556c020662a5a872b29117fa5cf375b7b886b238bb8aad73e1a08 WatchSource:0}: Error finding container 133c0a2af4f556c020662a5a872b29117fa5cf375b7b886b238bb8aad73e1a08: Status 404 returned error can't find the container with id 133c0a2af4f556c020662a5a872b29117fa5cf375b7b886b238bb8aad73e1a08
Jul 02 18:31:11.841210 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.841157    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:11.986756 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.986729    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 18:31:11.986944 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.986865    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-7d9mj" event={"ID":"f4e75b84-1c82-46c8-b414-f252d7ac5633","Type":"ContainerStarted","Data":"9078588f8c8255ca1670e18663694257874d7b9c34d01d8301775690ac24af82"}
Jul 02 18:31:11.987363 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.987296    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:31:11.988433 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.988402    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dp4vg" event={"ID":"572ae32e-0565-40c9-97eb-21161b4fb2c9","Type":"ContainerStarted","Data":"133c0a2af4f556c020662a5a872b29117fa5cf375b7b886b238bb8aad73e1a08"}
Jul 02 18:31:11.992681 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:11.992658    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-7d9mj"
Jul 02 18:31:12.003953 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:12.003872    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-7d9mj" podStartSLOduration=18.357919329 podStartE2EDuration="23.003857473s" totalImagesPullingTime="4.645938144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:12.003537981 +0000 UTC m=+64.073165528" watchObservedRunningTime="2026-07-02 18:31:12.003857473 +0000 UTC m=+64.073485000"
Jul 02 18:31:12.125222 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:12.125175    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:12.230069 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:12.230012    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:12.428183 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:12.428098    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:12.492507 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:12.492453    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:12.810142 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:12.810108    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:12.905932 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:12.905900    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:13.020660 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.020616    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:13.121429 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.121319    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:13.231910 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.231867    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:13.259222 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.259174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:31:13.259399 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.259298    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs\") pod \"network-metrics-daemon-m2h4c\" (UID: \"780da6ef-b5e6-43c9-b539-185cc5434c63\") " pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:31:13.262291 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.262248    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 18:31:13.262424 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.262250    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 18:31:13.272830 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.272798    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780da6ef-b5e6-43c9-b539-185cc5434c63-metrics-certs\") pod \"network-metrics-daemon-m2h4c\" (UID: \"780da6ef-b5e6-43c9-b539-185cc5434c63\") " pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:31:13.272830 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.272811    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3682ee4-82ef-4457-a048-4de940c25ac0-original-pull-secret\") pod \"global-pull-secret-syncer-5j4fb\" (UID: \"d3682ee4-82ef-4457-a048-4de940c25ac0\") " pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:31:13.287395 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.287370    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:13.296627 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.296598    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xq22w\""
Jul 02 18:31:13.298756 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.298732    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j4fb"
Jul 02 18:31:13.304227 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.304179    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2h4c"
Jul 02 18:31:13.360962 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.360520    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwj6\" (UniqueName: \"kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6\") pod \"network-check-target-b7jnx\" (UID: \"7340eb6c-cda8-42dc-9abe-2b16be84f537\") " pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:31:13.363873 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.363854    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 18:31:13.374450 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.374428    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 18:31:13.385075 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.385045    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxwj6\" (UniqueName: \"kubernetes.io/projected/7340eb6c-cda8-42dc-9abe-2b16be84f537-kube-api-access-sxwj6\") pod \"network-check-target-b7jnx\" (UID: \"7340eb6c-cda8-42dc-9abe-2b16be84f537\") " pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:31:13.429561 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.429501    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:13.460285 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.460261    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5j4fb"]
Jul 02 18:31:13.462264 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:31:13.462238    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3682ee4_82ef_4457_a048_4de940c25ac0.slice/crio-5b41658358130113b6ef816660d2fe05ddb5be13823b4e6924f1e54ee22dbf9a WatchSource:0}: Error finding container 5b41658358130113b6ef816660d2fe05ddb5be13823b4e6924f1e54ee22dbf9a: Status 404 returned error can't find the container with id 5b41658358130113b6ef816660d2fe05ddb5be13823b4e6924f1e54ee22dbf9a
Jul 02 18:31:13.485912 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.485889    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m2h4c"]
Jul 02 18:31:13.487959 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:31:13.487936    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780da6ef_b5e6_43c9_b539_185cc5434c63.slice/crio-4417479771b694a5dffde4660f0aea78aa5c3897b9ba03ac585392198a82462f WatchSource:0}: Error finding container 4417479771b694a5dffde4660f0aea78aa5c3897b9ba03ac585392198a82462f: Status 404 returned error can't find the container with id 4417479771b694a5dffde4660f0aea78aa5c3897b9ba03ac585392198a82462f
Jul 02 18:31:13.488523 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.488497    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:13.606725 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.606699    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pqjw5\""
Jul 02 18:31:13.613697 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.613674    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:31:13.614038 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.614010    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:13.714859 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.714824    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:13.817237 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.817186    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:13.997293 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.997220    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m2h4c" event={"ID":"780da6ef-b5e6-43c9-b539-185cc5434c63","Type":"ContainerStarted","Data":"4417479771b694a5dffde4660f0aea78aa5c3897b9ba03ac585392198a82462f"}
Jul 02 18:31:13.998479 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:13.998449    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5j4fb" event={"ID":"d3682ee4-82ef-4457-a048-4de940c25ac0","Type":"ContainerStarted","Data":"5b41658358130113b6ef816660d2fe05ddb5be13823b4e6924f1e54ee22dbf9a"}
Jul 02 18:31:14.044230 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:14.044172    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:15.435870 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:15.435784    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:15.535867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:15.535826    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:15.579866 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:15.579833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:31:15.580026 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:15.579882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:31:15.582887 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:15.582838    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls\") pod \"image-registry-596cd447fb-jh626\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") " pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:31:15.583520 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:15.583278    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/684a7ad0-005b-49d6-817b-b5bc7cbfb719-metrics-tls\") pod \"dns-default-9zq5n\" (UID: \"684a7ad0-005b-49d6-817b-b5bc7cbfb719\") " pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:31:15.827020 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:15.826989    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2gz4h\""
Jul 02 18:31:15.834447 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:15.834416    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:31:15.840003 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:15.839798    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cl9vj\""
Jul 02 18:31:15.847767 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:15.847520    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:31:16.105529 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:16.104177    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b7jnx"]
Jul 02 18:31:16.128686 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:16.128655    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9zq5n"]
Jul 02 18:31:16.148509 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:16.148480    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-596cd447fb-jh626"]
Jul 02 18:31:16.246742 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:31:16.246698    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7340eb6c_cda8_42dc_9abe_2b16be84f537.slice/crio-814b63806fb55440b3fe2a7bafaa69a439dc606448bb650a7bf1498a461282c1 WatchSource:0}: Error finding container 814b63806fb55440b3fe2a7bafaa69a439dc606448bb650a7bf1498a461282c1: Status 404 returned error can't find the container with id 814b63806fb55440b3fe2a7bafaa69a439dc606448bb650a7bf1498a461282c1
Jul 02 18:31:16.247093 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:31:16.247065    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684a7ad0_005b_49d6_817b_b5bc7cbfb719.slice/crio-a1b8cb246ab17bf3858a3e44b8fcb2d6b2f544f5fd9c52d3e002eae77f98086f WatchSource:0}: Error finding container a1b8cb246ab17bf3858a3e44b8fcb2d6b2f544f5fd9c52d3e002eae77f98086f: Status 404 returned error can't find the container with id a1b8cb246ab17bf3858a3e44b8fcb2d6b2f544f5fd9c52d3e002eae77f98086f
Jul 02 18:31:16.248852 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:31:16.248829    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded77eb69_1101_464b_9a74_e14bb394b7fa.slice/crio-acb42afdd6f4d75dd27841364502bec2710ae9e69bc6bfc1e781e377768ab6ad WatchSource:0}: Error finding container acb42afdd6f4d75dd27841364502bec2710ae9e69bc6bfc1e781e377768ab6ad: Status 404 returned error can't find the container with id acb42afdd6f4d75dd27841364502bec2710ae9e69bc6bfc1e781e377768ab6ad
Jul 02 18:31:17.026881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:17.026204    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m2h4c" event={"ID":"780da6ef-b5e6-43c9-b539-185cc5434c63","Type":"ContainerStarted","Data":"b365d4b6537e9a29e5ac880dac4f3336e910e675ffc3713f106092f071bc6feb"}
Jul 02 18:31:17.026881 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:17.026320    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m2h4c" event={"ID":"780da6ef-b5e6-43c9-b539-185cc5434c63","Type":"ContainerStarted","Data":"75e8b49119108bbb342f4be760cafb979fea9b24043826016facd4b8e182586c"}
Jul 02 18:31:17.028300 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:17.028263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b7jnx" event={"ID":"7340eb6c-cda8-42dc-9abe-2b16be84f537","Type":"ContainerStarted","Data":"814b63806fb55440b3fe2a7bafaa69a439dc606448bb650a7bf1498a461282c1"}
Jul 02 18:31:17.031069 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:17.031020    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9zq5n" event={"ID":"684a7ad0-005b-49d6-817b-b5bc7cbfb719","Type":"ContainerStarted","Data":"a1b8cb246ab17bf3858a3e44b8fcb2d6b2f544f5fd9c52d3e002eae77f98086f"}
Jul 02 18:31:17.033841 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:17.033815    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dp4vg" event={"ID":"572ae32e-0565-40c9-97eb-21161b4fb2c9","Type":"ContainerStarted","Data":"b8efb930766c316d86a31aec1fd2308aa133123d13f20eaf4d0cdb68820386cc"}
Jul 02 18:31:17.036720 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:17.036694    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596cd447fb-jh626" event={"ID":"ed77eb69-1101-464b-9a74-e14bb394b7fa","Type":"ContainerStarted","Data":"fedf1d7103bb40de689b53cf16fbaf303421e91e0d279f942134a041cabb7e0e"}
Jul 02 18:31:17.037106 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:17.037075    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596cd447fb-jh626" event={"ID":"ed77eb69-1101-464b-9a74-e14bb394b7fa","Type":"ContainerStarted","Data":"acb42afdd6f4d75dd27841364502bec2710ae9e69bc6bfc1e781e377768ab6ad"}
Jul 02 18:31:17.037375 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:17.037358    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:31:17.072947 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:17.072808    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-596cd447fb-jh626" podStartSLOduration=68.072789144 podStartE2EDuration="1m8.072789144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:17.071719526 +0000 UTC m=+69.141347055" watchObservedRunningTime="2026-07-02 18:31:17.072789144 +0000 UTC m=+69.142416673"
Jul 02 18:31:17.074096 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:17.074050    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m2h4c" podStartSLOduration=65.285492822 podStartE2EDuration="1m8.074034287s" totalImagesPullingTime="2.788541465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:17.045707329 +0000 UTC m=+69.115334857" watchObservedRunningTime="2026-07-02 18:31:17.074034287 +0000 UTC m=+69.143661815"
Jul 02 18:31:18.043174 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:18.043136    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dp4vg" event={"ID":"572ae32e-0565-40c9-97eb-21161b4fb2c9","Type":"ContainerStarted","Data":"6e434f36df0503341709b47c4491e9ef3a5b74701033bc6954e1cdabc25d6e7a"}
Jul 02 18:31:18.381678 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:18.381644    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:18.493610 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:18.493572    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:21.434152 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:21.433407    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:31:21.443216 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:21.438728    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b2c35b5-1bcf-4674-9b53-60abc6db4c07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gstlk\" (UID: \"5b2c35b5-1bcf-4674-9b53-60abc6db4c07\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:31:21.534921 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:21.534873    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:31:21.535104 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:21.535009    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:31:21.535824 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:21.535798    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7d790ff-691d-4fde-81c7-613493010723-service-ca-bundle\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:31:21.538374 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:21.538353    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d790ff-691d-4fde-81c7-613493010723-metrics-certs\") pod \"router-default-7f6f776dd5-qrrs8\" (UID: \"b7d790ff-691d-4fde-81c7-613493010723\") " pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:31:21.615506 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:21.615477    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-l44tr\""
Jul 02 18:31:21.622957 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:21.622928    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"
Jul 02 18:31:21.699432 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:21.699405    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2ggbw\""
Jul 02 18:31:21.707471 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:21.707443    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:31:21.751187 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:21.751162    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk"]
Jul 02 18:31:21.754098 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:31:21.754076    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2c35b5_1bcf_4674_9b53_60abc6db4c07.slice/crio-a9a6e2c30c514f8d2f8e777feae90aee8fd541d11f4d320dfadeb09688f46071 WatchSource:0}: Error finding container a9a6e2c30c514f8d2f8e777feae90aee8fd541d11f4d320dfadeb09688f46071: Status 404 returned error can't find the container with id a9a6e2c30c514f8d2f8e777feae90aee8fd541d11f4d320dfadeb09688f46071
Jul 02 18:31:21.842732 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:21.842528    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f6f776dd5-qrrs8"]
Jul 02 18:31:21.844968 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:31:21.844947    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d790ff_691d_4fde_81c7_613493010723.slice/crio-1327d659a0e8c33c3e5826be5076cfaf0e9804b8150e65f0dda0fe4abf542164 WatchSource:0}: Error finding container 1327d659a0e8c33c3e5826be5076cfaf0e9804b8150e65f0dda0fe4abf542164: Status 404 returned error can't find the container with id 1327d659a0e8c33c3e5826be5076cfaf0e9804b8150e65f0dda0fe4abf542164
Jul 02 18:31:22.060510 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.060473    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5j4fb" event={"ID":"d3682ee4-82ef-4457-a048-4de940c25ac0","Type":"ContainerStarted","Data":"69cc00ef67f916fa5a3499da8935e3da6a434c8cede02f01c11f905e47f2996e"}
Jul 02 18:31:22.061924 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.061894    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b7jnx" event={"ID":"7340eb6c-cda8-42dc-9abe-2b16be84f537","Type":"ContainerStarted","Data":"6ea524e658360f59e3af75b8fc108b0dcb3bca034aa6ab1ae5fc6c876d6169af"}
Jul 02 18:31:22.062071 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.061958    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:31:22.063447 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.063421    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f6f776dd5-qrrs8" event={"ID":"b7d790ff-691d-4fde-81c7-613493010723","Type":"ContainerStarted","Data":"3e170e78a3107f20d9ca76726a42b7635189684fa82c6e4f8f74d971dbcfaef3"}
Jul 02 18:31:22.063554 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.063472    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f6f776dd5-qrrs8" event={"ID":"b7d790ff-691d-4fde-81c7-613493010723","Type":"ContainerStarted","Data":"1327d659a0e8c33c3e5826be5076cfaf0e9804b8150e65f0dda0fe4abf542164"}
Jul 02 18:31:22.065306 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.065285    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9zq5n" event={"ID":"684a7ad0-005b-49d6-817b-b5bc7cbfb719","Type":"ContainerStarted","Data":"5bd2a5e5a4f5ae49edcdf0d25204a1a2f771dcce9692796148037cc9554f11b2"}
Jul 02 18:31:22.065573 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.065335    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:31:22.065573 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.065352    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9zq5n" event={"ID":"684a7ad0-005b-49d6-817b-b5bc7cbfb719","Type":"ContainerStarted","Data":"716a24f20f7f28120d6f2b4ab897b8b2288584dbced0bb4935d67464ba441678"}
Jul 02 18:31:22.067547 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.067525    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dp4vg" event={"ID":"572ae32e-0565-40c9-97eb-21161b4fb2c9","Type":"ContainerStarted","Data":"246dd5b5f6cf022348083e48160306d12138a01227b19bb7d2a62ca23c41f486"}
Jul 02 18:31:22.068685 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.068666    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk" event={"ID":"5b2c35b5-1bcf-4674-9b53-60abc6db4c07","Type":"ContainerStarted","Data":"a9a6e2c30c514f8d2f8e777feae90aee8fd541d11f4d320dfadeb09688f46071"}
Jul 02 18:31:22.100240 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.100118    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5j4fb" podStartSLOduration=65.227162757 podStartE2EDuration="1m13.100100437s" totalImagesPullingTime="7.87293768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:22.099570288 +0000 UTC m=+74.169197817" watchObservedRunningTime="2026-07-02 18:31:22.100100437 +0000 UTC m=+74.169727966"
Jul 02 18:31:22.133081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.133023    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7f6f776dd5-qrrs8" podStartSLOduration=33.13300442 podStartE2EDuration="33.13300442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:22.132876807 +0000 UTC m=+74.202504335" watchObservedRunningTime="2026-07-02 18:31:22.13300442 +0000 UTC m=+74.202631939"
Jul 02 18:31:22.156078 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.156027    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b7jnx" podStartSLOduration=68.069441857 podStartE2EDuration="1m13.156012524s" totalImagesPullingTime="5.086570667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:22.15579505 +0000 UTC m=+74.225422588" watchObservedRunningTime="2026-07-02 18:31:22.156012524 +0000 UTC m=+74.225640070"
Jul 02 18:31:22.177704 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.177664    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9zq5n" podStartSLOduration=34.092712235 podStartE2EDuration="39.177649651s" totalImagesPullingTime="5.084937416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:22.176660069 +0000 UTC m=+74.246287598" watchObservedRunningTime="2026-07-02 18:31:22.177649651 +0000 UTC m=+74.247277179"
Jul 02 18:31:22.199238 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.199167    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dp4vg" podStartSLOduration=17.768468337 podStartE2EDuration="27.199149373s" totalImagesPullingTime="9.430681036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:22.198544841 +0000 UTC m=+74.268172370" watchObservedRunningTime="2026-07-02 18:31:22.199149373 +0000 UTC m=+74.268776902"
Jul 02 18:31:22.510569 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.510419    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:22.631750 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.631716    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:22.708455 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.708417    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:31:22.711463 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.711437    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:31:22.749681 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.749648    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:22.904127 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.904095    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:22.976827 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:22.976737    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:23.073267 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:23.073234    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:31:23.073938 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:23.073920    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7f6f776dd5-qrrs8"
Jul 02 18:31:23.263848 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:23.263816    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:23.366369 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:23.366337    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:23.760421 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:23.760340    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:23.860516 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:23.860484    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:23.935646 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:23.935603    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:23.935847 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:23.935830    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2"]
Jul 02 18:31:23.938884 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:23.938867    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2"
Jul 02 18:31:23.941748 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:23.941723    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 02 18:31:23.941872 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:23.941725    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-546n4\""
Jul 02 18:31:23.953788 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:23.953762    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2"]
Jul 02 18:31:24.054803 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:24.054767    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/253f9a92-7d58-48c5-be09-36c69d2559e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-fd8v2\" (UID: \"253f9a92-7d58-48c5-be09-36c69d2559e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2"
Jul 02 18:31:24.077142 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:24.077101    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk" event={"ID":"5b2c35b5-1bcf-4674-9b53-60abc6db4c07","Type":"ContainerStarted","Data":"ef0c5de8b82829a88ea77e8399b3be93d280af8b87df82f2f01c6e94b802eb91"}
Jul 02 18:31:24.093283 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:24.093229    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gstlk" podStartSLOduration=33.411276808 podStartE2EDuration="35.09321295s" totalImagesPullingTime="1.681936142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:24.093091323 +0000 UTC m=+76.162718853" watchObservedRunningTime="2026-07-02 18:31:24.09321295 +0000 UTC m=+76.162840469"
Jul 02 18:31:24.155753 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:24.155714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/253f9a92-7d58-48c5-be09-36c69d2559e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-fd8v2\" (UID: \"253f9a92-7d58-48c5-be09-36c69d2559e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2"
Jul 02 18:31:24.155937 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:31:24.155833    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 02 18:31:24.155937 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:31:24.155899    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253f9a92-7d58-48c5-be09-36c69d2559e7-tls-certificates podName:253f9a92-7d58-48c5-be09-36c69d2559e7 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:24.6558799 +0000 UTC m=+76.725507405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/253f9a92-7d58-48c5-be09-36c69d2559e7-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-fd8v2" (UID: "253f9a92-7d58-48c5-be09-36c69d2559e7") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 02 18:31:24.659579 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:24.659543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/253f9a92-7d58-48c5-be09-36c69d2559e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-fd8v2\" (UID: \"253f9a92-7d58-48c5-be09-36c69d2559e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2"
Jul 02 18:31:24.659752 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:31:24.659647    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 02 18:31:24.659752 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:31:24.659703    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253f9a92-7d58-48c5-be09-36c69d2559e7-tls-certificates podName:253f9a92-7d58-48c5-be09-36c69d2559e7 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:25.65968835 +0000 UTC m=+77.729315855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/253f9a92-7d58-48c5-be09-36c69d2559e7-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-fd8v2" (UID: "253f9a92-7d58-48c5-be09-36c69d2559e7") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 02 18:31:25.141039 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:25.141002    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:25.669593 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:25.669553    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/253f9a92-7d58-48c5-be09-36c69d2559e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-fd8v2\" (UID: \"253f9a92-7d58-48c5-be09-36c69d2559e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2"
Jul 02 18:31:25.671913 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:25.671888    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/253f9a92-7d58-48c5-be09-36c69d2559e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-fd8v2\" (UID: \"253f9a92-7d58-48c5-be09-36c69d2559e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2"
Jul 02 18:31:25.754749 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:25.754713    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2"
Jul 02 18:31:25.908698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:25.908664    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2"]
Jul 02 18:31:25.912363 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:31:25.912336    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253f9a92_7d58_48c5_be09_36c69d2559e7.slice/crio-475390d1e2bbcceaaf4a053c1f52f7d1f1aaeaa40f169979d56fa52e44fe7370 WatchSource:0}: Error finding container 475390d1e2bbcceaaf4a053c1f52f7d1f1aaeaa40f169979d56fa52e44fe7370: Status 404 returned error can't find the container with id 475390d1e2bbcceaaf4a053c1f52f7d1f1aaeaa40f169979d56fa52e44fe7370
Jul 02 18:31:26.085253 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:26.085218    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2" event={"ID":"253f9a92-7d58-48c5-be09-36c69d2559e7","Type":"ContainerStarted","Data":"475390d1e2bbcceaaf4a053c1f52f7d1f1aaeaa40f169979d56fa52e44fe7370"}
Jul 02 18:31:26.915229 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:26.915166    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:27.017294 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:27.017253    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:27.114534 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:27.114506    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:27.162589 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:27.162539    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:28.094950 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:28.094912    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2" event={"ID":"253f9a92-7d58-48c5-be09-36c69d2559e7","Type":"ContainerStarted","Data":"5cd5a97fea64bf5ead43246e8396efc1e68090c2cc911a2a92e5d41edfe0a826"}
Jul 02 18:31:28.095369 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:28.095101    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2"
Jul 02 18:31:28.100060 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:28.100037    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2"
Jul 02 18:31:28.119844 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:28.119789    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-fd8v2" podStartSLOduration=3.6377948289999997 podStartE2EDuration="5.119771813s" totalImagesPullingTime="1.481976984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:28.119147443 +0000 UTC m=+80.188774973" watchObservedRunningTime="2026-07-02 18:31:28.119771813 +0000 UTC m=+80.189399341"
Jul 02 18:31:28.317721 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:28.317685    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:28.370668 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:28.370560    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:28.959444 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:28.959398    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:29.039108 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:29.039066    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:29.382955 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:29.382913    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:31.363403 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:31.363272    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:31.462942 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:31.462908    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:31.516291 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:31.516248    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:32.074851 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:32.074820    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9zq5n"
Jul 02 18:31:32.448985 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:32.448895    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:32.507981 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:32.507933    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:33.033395 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:33.033347    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:33.374674 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:33.374638    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:33.429723 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:33.429678    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:35.989390 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:35.989357    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:36.738112 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:36.738078    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:36.857272 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:36.857234    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:38.129917 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:38.129884    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:38.228570 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:38.228536    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:39.052322 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:39.052290    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:31:39.317127 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:39.317044    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:39.372962 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:39.372924    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:40.516343 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:40.516310    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:40.600739 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:40.600703    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:41.182409 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.182357    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:41.297776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.297733    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:41.401312 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.401280    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:41.503500 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.503418    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gt7jn"]
Jul 02 18:31:41.507413 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.507386    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.508181 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.508147    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:41.510231 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.510208    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 18:31:41.510391 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.510367    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 18:31:41.510391 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.510397    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 18:31:41.511943 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.511924    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 18:31:41.512076 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.511928    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mr82l\""
Jul 02 18:31:41.590759 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.590725    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:41.598251 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.598224    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-textfile\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.598402 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.598254    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thq8n\" (UniqueName: \"kubernetes.io/projected/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-kube-api-access-thq8n\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.598402 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.598285    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-metrics-client-ca\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.598402 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.598350    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-wtmp\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.598568 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.598451    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-tls\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.598568 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.598494    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-root\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.598568 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.598534    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-sys\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.598702 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.598579    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.598702 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.598607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-accelerators-collector-config\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.699010 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.698972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.699173 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.699016    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-accelerators-collector-config\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.699173 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.699036    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-textfile\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.699173 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.699146    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thq8n\" (UniqueName: \"kubernetes.io/projected/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-kube-api-access-thq8n\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.699410 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.699392    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-textfile\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.699562 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.699545    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-metrics-client-ca\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.699629 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.699584    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-wtmp\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.699681 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.699643    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-tls\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.699744 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:31:41.699730    2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 02 18:31:41.699807 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.699788    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-wtmp\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.699862 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.699787    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-root\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.699862 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.699730    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-accelerators-collector-config\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.699862 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:31:41.699843    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-tls podName:b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:42.199798296 +0000 UTC m=+94.269425805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-tls") pod "node-exporter-gt7jn" (UID: "b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95") : secret "node-exporter-tls" not found
Jul 02 18:31:41.700026 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.699858    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-root\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.700026 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.699932    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-sys\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.700026 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.700022    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-sys\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.700220 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.700150    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-metrics-client-ca\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.701519 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.701501    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.710061 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.710037    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thq8n\" (UniqueName: \"kubernetes.io/projected/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-kube-api-access-thq8n\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:41.714927 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.714898    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:41.817498 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.817467    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:41.915011 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:41.914979    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:42.014868 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:42.014832    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:42.115287 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:42.115174    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:42.205639 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:42.205600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-tls\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:42.207894 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:42.207870    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95-node-exporter-tls\") pod \"node-exporter-gt7jn\" (UID: \"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95\") " pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:42.216249 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:42.216219    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:42.417461 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:42.417371    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gt7jn"
Jul 02 18:31:42.427681 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:31:42.427646    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b64b2d_5cef_4ba4_b19e_d2471cdf4f95.slice/crio-04d3be7562a0c6f45eaabc5a39ebde031dd753e48ca20dbd9cb30dd5a8e97ab2 WatchSource:0}: Error finding container 04d3be7562a0c6f45eaabc5a39ebde031dd753e48ca20dbd9cb30dd5a8e97ab2: Status 404 returned error can't find the container with id 04d3be7562a0c6f45eaabc5a39ebde031dd753e48ca20dbd9cb30dd5a8e97ab2
Jul 02 18:31:42.437675 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:42.437644    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:42.556883 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:42.556845    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:42.760023 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:42.759930    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:43.148348 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:43.148299    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gt7jn" event={"ID":"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95","Type":"ContainerStarted","Data":"04d3be7562a0c6f45eaabc5a39ebde031dd753e48ca20dbd9cb30dd5a8e97ab2"}
Jul 02 18:31:43.612893 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:43.612860    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:43.717277 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:43.713855    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:44.153666 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:44.153633    2575 generic.go:356] "Generic (PLEG): container finished" podID="b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95" containerID="c79517aa61a67c4067730762074dfbfdd90f97dc9585857a2d29673ac77e11c0" exitCode=0
Jul 02 18:31:44.153666 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:44.153671    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gt7jn" event={"ID":"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95","Type":"ContainerDied","Data":"c79517aa61a67c4067730762074dfbfdd90f97dc9585857a2d29673ac77e11c0"}
Jul 02 18:31:44.434502 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:44.434411    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:44.537160 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:44.537125    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:44.970712 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:44.970675    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:45.110876 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:45.110846    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:45.159356 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:45.159318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gt7jn" event={"ID":"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95","Type":"ContainerStarted","Data":"b84a4af9e4e098cb393e0efcf00b6329d3f57458c340ffaee186f7ea9f71778f"}
Jul 02 18:31:45.159762 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:45.159373    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gt7jn" event={"ID":"b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95","Type":"ContainerStarted","Data":"56ebd63418400b4852dcc5654dfec01863f7f13045cb0072d7e39ba1d8c9fd15"}
Jul 02 18:31:45.184638 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:45.184594    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gt7jn" podStartSLOduration=3.164307993 podStartE2EDuration="4.184580191s" totalImagesPullingTime="954.308257ms" totalInitContainerRuntime="65.963941ms" isStatefulPod=true podCreationTimestamp="2026-07-02 18:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:45.183226523 +0000 UTC m=+97.252854050" watchObservedRunningTime="2026-07-02 18:31:45.184580191 +0000 UTC m=+97.254207719"
Jul 02 18:31:45.826360 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:45.826325    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-596cd447fb-jh626"]
Jul 02 18:31:46.017159 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:46.017122    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:46.117629 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:46.117532    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:46.245957 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:46.245920    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:46.346966 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:46.346938    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:47.247777 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.247740    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:47.780252 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.780220    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 18:31:47.780834 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.780802    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:47.783203 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.783177    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.791832 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.791813    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 02 18:31:47.791949 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.791836    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 02 18:31:47.791949 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.791836    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 02 18:31:47.792139 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.792121    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 02 18:31:47.792206 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.792152    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 02 18:31:47.792206 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.792153    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 02 18:31:47.792312 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.792153    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 02 18:31:47.792312 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.792237    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 02 18:31:47.792416 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.792404    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 02 18:31:47.793433 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.793419    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 02 18:31:47.793476 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.793449    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-w2vw2\""
Jul 02 18:31:47.793734 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.793718    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 02 18:31:47.793773 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.793733    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 02 18:31:47.793872 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.793854    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7huuabgn4q6ka\""
Jul 02 18:31:47.794120 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.794108    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 02 18:31:47.800822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.800795    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 02 18:31:47.802105 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.802088    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 02 18:31:47.807541 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.807517    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 18:31:47.952834 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.952802    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01f465e2-fe9b-49ce-a4cc-849e78e19611-config-out\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.952984 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.952845    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.952984 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.952918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.952984 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.952960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953093 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.952988    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953093 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953093 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953044    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953093 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953067    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01f465e2-fe9b-49ce-a4cc-849e78e19611-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953093 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953364 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953108    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953364 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953180    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953364 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953259    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-config\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953364 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953289    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953364 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953315    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953364 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953355    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953619 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953377    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953619 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953391    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-web-config\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953619 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953437    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt2wr\" (UniqueName: \"kubernetes.io/projected/01f465e2-fe9b-49ce-a4cc-849e78e19611-kube-api-access-nt2wr\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953619 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953464    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:47.953619 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:47.953498    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054138 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054105    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054329 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054150    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01f465e2-fe9b-49ce-a4cc-849e78e19611-config-out\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054329 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054185    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054329 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054239    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054329 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054261    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054329 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054329 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054633 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054349    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054633 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01f465e2-fe9b-49ce-a4cc-849e78e19611-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054633 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054407    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054633 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054633 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054633 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054519    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-config\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054633 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054633 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054570    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054633 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.054633 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054628    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.055122 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054653    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-web-config\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.055122 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt2wr\" (UniqueName: \"kubernetes.io/projected/01f465e2-fe9b-49ce-a4cc-849e78e19611-kube-api-access-nt2wr\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.055122 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.055122 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.054740    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.055122 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.055014    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.055581 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.055556    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.055783 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.055758    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.056976 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.056954    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.057226 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.057175    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.057897 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.057878    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.058036 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.057993    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.058679 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.058656    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.058808 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.058670    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.059160 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.059137    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01f465e2-fe9b-49ce-a4cc-849e78e19611-config-out\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.059273 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.059231    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.059273 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.059231    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.059451 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.059433    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.060104 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.060079    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-web-config\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.060186 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.060125    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.060493 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.060475    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01f465e2-fe9b-49ce-a4cc-849e78e19611-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.060553 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.060497    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-config\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.060878 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.060856    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.066483 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.066463    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt2wr\" (UniqueName: \"kubernetes.io/projected/01f465e2-fe9b-49ce-a4cc-849e78e19611-kube-api-access-nt2wr\") pod \"prometheus-k8s-0\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.092591 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.092567    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:31:48.231327 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.231301    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 18:31:48.233475 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:31:48.233443    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f465e2_fe9b_49ce_a4cc_849e78e19611.slice/crio-01b2135c0eca02788ccf004743d34f98b0deb88509b9d59c0a2d3c6279b7e7ea WatchSource:0}: Error finding container 01b2135c0eca02788ccf004743d34f98b0deb88509b9d59c0a2d3c6279b7e7ea: Status 404 returned error can't find the container with id 01b2135c0eca02788ccf004743d34f98b0deb88509b9d59c0a2d3c6279b7e7ea
Jul 02 18:31:48.381645 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.381563    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:48.472337 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:48.472295    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:49.182939 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:49.182885    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerStarted","Data":"01b2135c0eca02788ccf004743d34f98b0deb88509b9d59c0a2d3c6279b7e7ea"}
Jul 02 18:31:49.232549 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:49.232516    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:49.332483 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:49.332453    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:50.187716 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:50.187679    2575 generic.go:356] "Generic (PLEG): container finished" podID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerID="e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0" exitCode=0
Jul 02 18:31:50.188098 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:50.187760    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerDied","Data":"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"}
Jul 02 18:31:51.861202 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:51.861171    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:51.944646 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:51.944607    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:52.009010 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:52.008940    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:53.075493 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:53.075463    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b7jnx"
Jul 02 18:31:53.160550 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:53.160497    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:53.234065 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:53.234026    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:54.686366 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:54.686330    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:54.755959 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:54.755921    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:55.213352 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:55.213314    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerStarted","Data":"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"}
Jul 02 18:31:55.213538 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:55.213403    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerStarted","Data":"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"}
Jul 02 18:31:56.883183 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:56.883155    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:56.985319 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:56.985290    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:57.086304 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:57.086278    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:57.225351 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:57.225323    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerStarted","Data":"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"}
Jul 02 18:31:57.225436 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:57.225419    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerStarted","Data":"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"}
Jul 02 18:31:57.225486 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:57.225452    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerStarted","Data":"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"}
Jul 02 18:31:58.233071 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:58.233035    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerStarted","Data":"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"}
Jul 02 18:31:58.275488 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:31:58.275442    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.946396848 podStartE2EDuration="11.275426787s" totalImagesPullingTime="7.307675413s" totalInitContainerRuntime="21.354526ms" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:58.274596637 +0000 UTC m=+110.344224167" watchObservedRunningTime="2026-07-02 18:31:58.275426787 +0000 UTC m=+110.345054315"
Jul 02 18:32:01.244823 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:01.244785    2575 generic.go:356] "Generic (PLEG): container finished" podID="6846ce7e-056b-4aad-88cd-572cf4d80df3" containerID="df207a9ca03715d2c08d500bd5bc5ff82ba8320fd83ac9ecc44cfe53b797af1e" exitCode=0
Jul 02 18:32:01.245236 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:01.244857    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-w5w95" event={"ID":"6846ce7e-056b-4aad-88cd-572cf4d80df3","Type":"ContainerDied","Data":"df207a9ca03715d2c08d500bd5bc5ff82ba8320fd83ac9ecc44cfe53b797af1e"}
Jul 02 18:32:01.245236 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:01.245215    2575 scope.go:117] "RemoveContainer" containerID="df207a9ca03715d2c08d500bd5bc5ff82ba8320fd83ac9ecc44cfe53b797af1e"
Jul 02 18:32:02.001326 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:02.001280    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:32:02.103405 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:02.103362    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:32:02.251970 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:02.251882    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-w5w95" event={"ID":"6846ce7e-056b-4aad-88cd-572cf4d80df3","Type":"ContainerStarted","Data":"a8a17fd15e599694378c92c7d64e0ce782c7cf990a0e91510e90d7859271d146"}
Jul 02 18:32:03.093586 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:03.093547    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:32:03.381609 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:03.381507    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:32:03.480967 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:03.480929    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:32:03.923388 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:03.923349    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:32:04.022922 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:04.022866    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:32:04.123948 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:04.123909    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:32:10.849007 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:10.848946    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-596cd447fb-jh626" podUID="ed77eb69-1101-464b-9a74-e14bb394b7fa" containerName="registry" containerID="cri-o://fedf1d7103bb40de689b53cf16fbaf303421e91e0d279f942134a041cabb7e0e" gracePeriod=30
Jul 02 18:32:11.100067 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.100005    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:32:11.164990 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.164961    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed77eb69-1101-464b-9a74-e14bb394b7fa-ca-trust-extracted\") pod \"ed77eb69-1101-464b-9a74-e14bb394b7fa\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") "
Jul 02 18:32:11.165147 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.165001    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed77eb69-1101-464b-9a74-e14bb394b7fa-image-registry-private-configuration\") pod \"ed77eb69-1101-464b-9a74-e14bb394b7fa\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") "
Jul 02 18:32:11.165147 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.165032    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed77eb69-1101-464b-9a74-e14bb394b7fa-trusted-ca\") pod \"ed77eb69-1101-464b-9a74-e14bb394b7fa\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") "
Jul 02 18:32:11.165147 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.165079    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-certificates\") pod \"ed77eb69-1101-464b-9a74-e14bb394b7fa\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") "
Jul 02 18:32:11.165147 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.165104    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58cq7\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-kube-api-access-58cq7\") pod \"ed77eb69-1101-464b-9a74-e14bb394b7fa\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") "
Jul 02 18:32:11.165147 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.165142    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls\") pod \"ed77eb69-1101-464b-9a74-e14bb394b7fa\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") "
Jul 02 18:32:11.165499 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.165181    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed77eb69-1101-464b-9a74-e14bb394b7fa-installation-pull-secrets\") pod \"ed77eb69-1101-464b-9a74-e14bb394b7fa\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") "
Jul 02 18:32:11.165499 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.165239    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-bound-sa-token\") pod \"ed77eb69-1101-464b-9a74-e14bb394b7fa\" (UID: \"ed77eb69-1101-464b-9a74-e14bb394b7fa\") "
Jul 02 18:32:11.165608 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.165546    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ed77eb69-1101-464b-9a74-e14bb394b7fa" (UID: "ed77eb69-1101-464b-9a74-e14bb394b7fa"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:32:11.165608 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.165567    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed77eb69-1101-464b-9a74-e14bb394b7fa-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ed77eb69-1101-464b-9a74-e14bb394b7fa" (UID: "ed77eb69-1101-464b-9a74-e14bb394b7fa"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:32:11.167759 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.167731    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed77eb69-1101-464b-9a74-e14bb394b7fa-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ed77eb69-1101-464b-9a74-e14bb394b7fa" (UID: "ed77eb69-1101-464b-9a74-e14bb394b7fa"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:32:11.167922 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.167893    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ed77eb69-1101-464b-9a74-e14bb394b7fa" (UID: "ed77eb69-1101-464b-9a74-e14bb394b7fa"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:32:11.168045 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.167938    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ed77eb69-1101-464b-9a74-e14bb394b7fa" (UID: "ed77eb69-1101-464b-9a74-e14bb394b7fa"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:32:11.168045 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.168007    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed77eb69-1101-464b-9a74-e14bb394b7fa-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ed77eb69-1101-464b-9a74-e14bb394b7fa" (UID: "ed77eb69-1101-464b-9a74-e14bb394b7fa"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:32:11.168123 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.168084    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-kube-api-access-58cq7" (OuterVolumeSpecName: "kube-api-access-58cq7") pod "ed77eb69-1101-464b-9a74-e14bb394b7fa" (UID: "ed77eb69-1101-464b-9a74-e14bb394b7fa"). InnerVolumeSpecName "kube-api-access-58cq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:32:11.173323 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.173298    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed77eb69-1101-464b-9a74-e14bb394b7fa-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ed77eb69-1101-464b-9a74-e14bb394b7fa" (UID: "ed77eb69-1101-464b-9a74-e14bb394b7fa"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:32:11.266248 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.266208    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed77eb69-1101-464b-9a74-e14bb394b7fa-ca-trust-extracted\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:32:11.266248 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.266242    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed77eb69-1101-464b-9a74-e14bb394b7fa-image-registry-private-configuration\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:32:11.266248 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.266252    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed77eb69-1101-464b-9a74-e14bb394b7fa-trusted-ca\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:32:11.266466 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.266262    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-certificates\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:32:11.266466 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.266271    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58cq7\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-kube-api-access-58cq7\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:32:11.266466 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.266281    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-registry-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:32:11.266466 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.266290    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed77eb69-1101-464b-9a74-e14bb394b7fa-installation-pull-secrets\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:32:11.266466 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.266299    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed77eb69-1101-464b-9a74-e14bb394b7fa-bound-sa-token\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:32:11.290286 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.290253    2575 generic.go:356] "Generic (PLEG): container finished" podID="ed77eb69-1101-464b-9a74-e14bb394b7fa" containerID="fedf1d7103bb40de689b53cf16fbaf303421e91e0d279f942134a041cabb7e0e" exitCode=0
Jul 02 18:32:11.290412 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.290304    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596cd447fb-jh626"
Jul 02 18:32:11.290412 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.290344    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596cd447fb-jh626" event={"ID":"ed77eb69-1101-464b-9a74-e14bb394b7fa","Type":"ContainerDied","Data":"fedf1d7103bb40de689b53cf16fbaf303421e91e0d279f942134a041cabb7e0e"}
Jul 02 18:32:11.290412 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.290397    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596cd447fb-jh626" event={"ID":"ed77eb69-1101-464b-9a74-e14bb394b7fa","Type":"ContainerDied","Data":"acb42afdd6f4d75dd27841364502bec2710ae9e69bc6bfc1e781e377768ab6ad"}
Jul 02 18:32:11.290537 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.290419    2575 scope.go:117] "RemoveContainer" containerID="fedf1d7103bb40de689b53cf16fbaf303421e91e0d279f942134a041cabb7e0e"
Jul 02 18:32:11.300907 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.300879    2575 scope.go:117] "RemoveContainer" containerID="fedf1d7103bb40de689b53cf16fbaf303421e91e0d279f942134a041cabb7e0e"
Jul 02 18:32:11.301245 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:32:11.301216    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fedf1d7103bb40de689b53cf16fbaf303421e91e0d279f942134a041cabb7e0e\": container with ID starting with fedf1d7103bb40de689b53cf16fbaf303421e91e0d279f942134a041cabb7e0e not found: ID does not exist" containerID="fedf1d7103bb40de689b53cf16fbaf303421e91e0d279f942134a041cabb7e0e"
Jul 02 18:32:11.301296 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.301258    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fedf1d7103bb40de689b53cf16fbaf303421e91e0d279f942134a041cabb7e0e"} err="failed to get container status \"fedf1d7103bb40de689b53cf16fbaf303421e91e0d279f942134a041cabb7e0e\": rpc error: code = NotFound desc = could not find container \"fedf1d7103bb40de689b53cf16fbaf303421e91e0d279f942134a041cabb7e0e\": container with ID starting with fedf1d7103bb40de689b53cf16fbaf303421e91e0d279f942134a041cabb7e0e not found: ID does not exist"
Jul 02 18:32:11.313302 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.313273    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-596cd447fb-jh626"]
Jul 02 18:32:11.317055 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:11.317029    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-596cd447fb-jh626"]
Jul 02 18:32:12.488385 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:12.488346    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed77eb69-1101-464b-9a74-e14bb394b7fa" path="/var/lib/kubelet/pods/ed77eb69-1101-464b-9a74-e14bb394b7fa/volumes"
Jul 02 18:32:16.317205 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:16.317152    2575 generic.go:356] "Generic (PLEG): container finished" podID="cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f" containerID="07f6fcd2563d39936b5b8b90bf3cc2ff3e948ece52a584c7c35b8b7f29091d4b" exitCode=0
Jul 02 18:32:16.317634 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:16.317230    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs" event={"ID":"cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f","Type":"ContainerDied","Data":"07f6fcd2563d39936b5b8b90bf3cc2ff3e948ece52a584c7c35b8b7f29091d4b"}
Jul 02 18:32:16.317634 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:16.317528    2575 scope.go:117] "RemoveContainer" containerID="07f6fcd2563d39936b5b8b90bf3cc2ff3e948ece52a584c7c35b8b7f29091d4b"
Jul 02 18:32:17.324142 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:17.324097    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-8bvjs" event={"ID":"cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f","Type":"ContainerStarted","Data":"2c26ab0f06e87cea4da64e8129483b589248561581ebd77d99c3df8ef624711e"}
Jul 02 18:32:18.385429 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:18.385397    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:32:18.486663 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:18.486625    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:32:26.361265 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:26.361230    2575 generic.go:356] "Generic (PLEG): container finished" podID="28950053-fd3d-4df3-aae5-a38cbdda059d" containerID="04e8861f3a2335f6c2b7a36f8593d8a28c61b456d8b5f2eb1f1fcdcede86ee46" exitCode=0
Jul 02 18:32:26.361751 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:26.361281    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn" event={"ID":"28950053-fd3d-4df3-aae5-a38cbdda059d","Type":"ContainerDied","Data":"04e8861f3a2335f6c2b7a36f8593d8a28c61b456d8b5f2eb1f1fcdcede86ee46"}
Jul 02 18:32:26.361751 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:26.361662    2575 scope.go:117] "RemoveContainer" containerID="04e8861f3a2335f6c2b7a36f8593d8a28c61b456d8b5f2eb1f1fcdcede86ee46"
Jul 02 18:32:27.366027 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:27.365994    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-n5jkn" event={"ID":"28950053-fd3d-4df3-aae5-a38cbdda059d","Type":"ContainerStarted","Data":"52b48c3c9030481959ad5a0b95f9fca0777ce9a14f51f5c4ea2d21f9681ca93f"}
Jul 02 18:32:48.093818 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:48.093767    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:32:48.109982 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:48.109953    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:32:48.461174 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:48.461095    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:32:57.573723 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:57.573690    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:32:57.627910 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:32:57.627876    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:06.072693 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.072607    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 18:33:06.073290 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.072978    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="prometheus" containerID="cri-o://a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da" gracePeriod=600
Jul 02 18:33:06.073290 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.073022    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="config-reloader" containerID="cri-o://73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde" gracePeriod=600
Jul 02 18:33:06.073290 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.073037    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14" gracePeriod=600
Jul 02 18:33:06.073290 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.073084    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="kube-rbac-proxy" containerID="cri-o://3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf" gracePeriod=600
Jul 02 18:33:06.073290 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.073039    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="thanos-sidecar" containerID="cri-o://263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152" gracePeriod=600
Jul 02 18:33:06.073290 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.073022    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="kube-rbac-proxy-web" containerID="cri-o://e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14" gracePeriod=600
Jul 02 18:33:06.331173 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.331107    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.403101 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403073    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-config\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.403290 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403107    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-thanos-prometheus-http-client-file\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.403290 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403141    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-trusted-ca-bundle\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.403290 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403165    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt2wr\" (UniqueName: \"kubernetes.io/projected/01f465e2-fe9b-49ce-a4cc-849e78e19611-kube-api-access-nt2wr\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.403290 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403276    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-metrics-client-ca\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.403559 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403308    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-1\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.403559 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403333    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-metrics-client-certs\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.403559 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403358    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-kube-rbac-proxy\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.403559 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403400    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.403559 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403428    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-db\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.403559 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403452    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-kubelet-serving-ca-bundle\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.403559 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403477    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-web-config\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.403559 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403538    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.404297 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403591    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:33:06.404297 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403884    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:33:06.404297 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.403933    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.404713    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-serving-certs-ca-bundle\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.404721    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.404765    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01f465e2-fe9b-49ce-a4cc-849e78e19611-tls-assets\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.404787    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.404807    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-2\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.404845    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-0\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.404886    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01f465e2-fe9b-49ce-a4cc-849e78e19611-config-out\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.405008    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-grpc-tls\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.405041    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-tls\") pod \"01f465e2-fe9b-49ce-a4cc-849e78e19611\" (UID: \"01f465e2-fe9b-49ce-a4cc-849e78e19611\") "
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.405336    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-metrics-client-ca\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.405347    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.405356    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.405394    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-db\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.405414    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.405433    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.405822 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.405777    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:33:06.406897 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.406568    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f465e2-fe9b-49ce-a4cc-849e78e19611-kube-api-access-nt2wr" (OuterVolumeSpecName: "kube-api-access-nt2wr") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "kube-api-access-nt2wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:33:06.407054 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.407028    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-config" (OuterVolumeSpecName: "config") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:33:06.407316 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.407280    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:33:06.407494 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.407467    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:33:06.407569 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.407527    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:33:06.407710 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.407675    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:33:06.408523 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.408484    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:33:06.408856 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.408819    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:33:06.409465 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.409440    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f465e2-fe9b-49ce-a4cc-849e78e19611-config-out" (OuterVolumeSpecName: "config-out") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:33:06.409572 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.409518    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f465e2-fe9b-49ce-a4cc-849e78e19611-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:33:06.409628 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.409552    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:33:06.409898 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.409876    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:33:06.418777 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.418742    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-web-config" (OuterVolumeSpecName: "web-config") pod "01f465e2-fe9b-49ce-a4cc-849e78e19611" (UID: "01f465e2-fe9b-49ce-a4cc-849e78e19611"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:33:06.506552 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506521    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.506552 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506549    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.506552 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506559    2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01f465e2-fe9b-49ce-a4cc-849e78e19611-config-out\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.506776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506569    2575 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-grpc-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.506776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506583    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.506776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506593    2575 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.506776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506603    2575 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.506776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506612    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt2wr\" (UniqueName: \"kubernetes.io/projected/01f465e2-fe9b-49ce-a4cc-849e78e19611-kube-api-access-nt2wr\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.506776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506620    2575 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-metrics-client-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.506776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506629    2575 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-kube-rbac-proxy\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.506776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506637    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.506776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506671    2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-web-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.506776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506683    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01f465e2-fe9b-49ce-a4cc-849e78e19611-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.506776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506691    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f465e2-fe9b-49ce-a4cc-849e78e19611-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.506776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.506704    2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01f465e2-fe9b-49ce-a4cc-849e78e19611-tls-assets\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:33:06.514775 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.514736    2575 generic.go:356] "Generic (PLEG): container finished" podID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerID="1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14" exitCode=0
Jul 02 18:33:06.514775 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.514775    2575 generic.go:356] "Generic (PLEG): container finished" podID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerID="3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf" exitCode=0
Jul 02 18:33:06.514775 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.514786    2575 generic.go:356] "Generic (PLEG): container finished" podID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerID="e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14" exitCode=0
Jul 02 18:33:06.515001 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.514795    2575 generic.go:356] "Generic (PLEG): container finished" podID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerID="263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152" exitCode=0
Jul 02 18:33:06.515001 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.514805    2575 generic.go:356] "Generic (PLEG): container finished" podID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerID="73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde" exitCode=0
Jul 02 18:33:06.515001 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.514813    2575 generic.go:356] "Generic (PLEG): container finished" podID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerID="a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da" exitCode=0
Jul 02 18:33:06.515001 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.514813    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerDied","Data":"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"}
Jul 02 18:33:06.515001 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.514848    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.515001 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.514881    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerDied","Data":"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"}
Jul 02 18:33:06.515001 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.514909    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerDied","Data":"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"}
Jul 02 18:33:06.515001 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.514933    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerDied","Data":"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"}
Jul 02 18:33:06.515001 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.514947    2575 scope.go:117] "RemoveContainer" containerID="1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"
Jul 02 18:33:06.515001 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.514957    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerDied","Data":"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"}
Jul 02 18:33:06.515001 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.514983    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerDied","Data":"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"}
Jul 02 18:33:06.515453 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.515006    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f465e2-fe9b-49ce-a4cc-849e78e19611","Type":"ContainerDied","Data":"01b2135c0eca02788ccf004743d34f98b0deb88509b9d59c0a2d3c6279b7e7ea"}
Jul 02 18:33:06.523333 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.523294    2575 scope.go:117] "RemoveContainer" containerID="3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"
Jul 02 18:33:06.536207 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.536174    2575 scope.go:117] "RemoveContainer" containerID="e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"
Jul 02 18:33:06.537156 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.537134    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 18:33:06.541552 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.541526    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 18:33:06.544803 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.544786    2575 scope.go:117] "RemoveContainer" containerID="263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"
Jul 02 18:33:06.551819 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.551803    2575 scope.go:117] "RemoveContainer" containerID="73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"
Jul 02 18:33:06.558647 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.558619    2575 scope.go:117] "RemoveContainer" containerID="a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"
Jul 02 18:33:06.566019 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.565997    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 18:33:06.566310 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566294    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="thanos-sidecar"
Jul 02 18:33:06.566310 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566310    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="thanos-sidecar"
Jul 02 18:33:06.566423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566317    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="kube-rbac-proxy"
Jul 02 18:33:06.566423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566322    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="kube-rbac-proxy"
Jul 02 18:33:06.566423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566330    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="init-config-reloader"
Jul 02 18:33:06.566423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566336    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="init-config-reloader"
Jul 02 18:33:06.566423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566342    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="prometheus"
Jul 02 18:33:06.566423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566347    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="prometheus"
Jul 02 18:33:06.566423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566356    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="config-reloader"
Jul 02 18:33:06.566423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566364    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="config-reloader"
Jul 02 18:33:06.566423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566376    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed77eb69-1101-464b-9a74-e14bb394b7fa" containerName="registry"
Jul 02 18:33:06.566423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566383    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed77eb69-1101-464b-9a74-e14bb394b7fa" containerName="registry"
Jul 02 18:33:06.566423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566394    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="kube-rbac-proxy-thanos"
Jul 02 18:33:06.566423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566402    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="kube-rbac-proxy-thanos"
Jul 02 18:33:06.566423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566423    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="kube-rbac-proxy-web"
Jul 02 18:33:06.566867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566432    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="kube-rbac-proxy-web"
Jul 02 18:33:06.566867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566454    2575 scope.go:117] "RemoveContainer" containerID="e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"
Jul 02 18:33:06.566867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566480    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed77eb69-1101-464b-9a74-e14bb394b7fa" containerName="registry"
Jul 02 18:33:06.566867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566488    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="config-reloader"
Jul 02 18:33:06.566867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566494    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="kube-rbac-proxy-thanos"
Jul 02 18:33:06.566867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566500    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="thanos-sidecar"
Jul 02 18:33:06.566867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566505    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="kube-rbac-proxy"
Jul 02 18:33:06.566867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566510    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="kube-rbac-proxy-web"
Jul 02 18:33:06.566867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.566520    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" containerName="prometheus"
Jul 02 18:33:06.569902 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.569885    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.572828 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.572806    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 02 18:33:06.573030 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.573013    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 02 18:33:06.573117 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.573101    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 02 18:33:06.573202 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.573174    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 02 18:33:06.573259 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.573245    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 02 18:33:06.573317 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.573298    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 02 18:33:06.573378 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.573367    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-w2vw2\""
Jul 02 18:33:06.573636 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.573612    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7huuabgn4q6ka\""
Jul 02 18:33:06.573636 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.573644    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 02 18:33:06.573798 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.573692    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 02 18:33:06.573798 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.573780    2575 scope.go:117] "RemoveContainer" containerID="1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"
Jul 02 18:33:06.573939 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.573804    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 02 18:33:06.574105 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:33:06.574084    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14\": container with ID starting with 1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14 not found: ID does not exist" containerID="1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"
Jul 02 18:33:06.574174 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.574118    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"} err="failed to get container status \"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14\": rpc error: code = NotFound desc = could not find container \"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14\": container with ID starting with 1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14 not found: ID does not exist"
Jul 02 18:33:06.574174 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.574144    2575 scope.go:117] "RemoveContainer" containerID="3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"
Jul 02 18:33:06.574336 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.574322    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 02 18:33:06.574468 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:33:06.574447    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf\": container with ID starting with 3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf not found: ID does not exist" containerID="3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"
Jul 02 18:33:06.574511 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.574478    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"} err="failed to get container status \"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf\": rpc error: code = NotFound desc = could not find container \"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf\": container with ID starting with 3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf not found: ID does not exist"
Jul 02 18:33:06.574511 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.574501    2575 scope.go:117] "RemoveContainer" containerID="e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"
Jul 02 18:33:06.574765 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:33:06.574716    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14\": container with ID starting with e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14 not found: ID does not exist" containerID="e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"
Jul 02 18:33:06.574765 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.574757    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 02 18:33:06.574885 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.574760    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"} err="failed to get container status \"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14\": rpc error: code = NotFound desc = could not find container \"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14\": container with ID starting with e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14 not found: ID does not exist"
Jul 02 18:33:06.574885 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.574780    2575 scope.go:117] "RemoveContainer" containerID="263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"
Jul 02 18:33:06.574988 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.574933    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 02 18:33:06.575039 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.574994    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 02 18:33:06.575089 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:33:06.575064    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152\": container with ID starting with 263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152 not found: ID does not exist" containerID="263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"
Jul 02 18:33:06.575142 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.575091    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"} err="failed to get container status \"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152\": rpc error: code = NotFound desc = could not find container \"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152\": container with ID starting with 263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152 not found: ID does not exist"
Jul 02 18:33:06.575142 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.575108    2575 scope.go:117] "RemoveContainer" containerID="73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"
Jul 02 18:33:06.575432 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:33:06.575403    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde\": container with ID starting with 73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde not found: ID does not exist" containerID="73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"
Jul 02 18:33:06.575539 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.575439    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"} err="failed to get container status \"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde\": rpc error: code = NotFound desc = could not find container \"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde\": container with ID starting with 73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde not found: ID does not exist"
Jul 02 18:33:06.575539 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.575461    2575 scope.go:117] "RemoveContainer" containerID="a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"
Jul 02 18:33:06.575746 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:33:06.575722    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da\": container with ID starting with a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da not found: ID does not exist" containerID="a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"
Jul 02 18:33:06.575827 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.575748    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"} err="failed to get container status \"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da\": rpc error: code = NotFound desc = could not find container \"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da\": container with ID starting with a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da not found: ID does not exist"
Jul 02 18:33:06.575827 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.575761    2575 scope.go:117] "RemoveContainer" containerID="e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"
Jul 02 18:33:06.576061 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:33:06.576010    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0\": container with ID starting with e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0 not found: ID does not exist" containerID="e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"
Jul 02 18:33:06.576061 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.576042    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"} err="failed to get container status \"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0\": rpc error: code = NotFound desc = could not find container \"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0\": container with ID starting with e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0 not found: ID does not exist"
Jul 02 18:33:06.576305 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.576062    2575 scope.go:117] "RemoveContainer" containerID="1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"
Jul 02 18:33:06.576424 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.576399    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"} err="failed to get container status \"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14\": rpc error: code = NotFound desc = could not find container \"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14\": container with ID starting with 1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14 not found: ID does not exist"
Jul 02 18:33:06.576490 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.576428    2575 scope.go:117] "RemoveContainer" containerID="3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"
Jul 02 18:33:06.576740 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.576706    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"} err="failed to get container status \"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf\": rpc error: code = NotFound desc = could not find container \"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf\": container with ID starting with 3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf not found: ID does not exist"
Jul 02 18:33:06.576740 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.576735    2575 scope.go:117] "RemoveContainer" containerID="e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"
Jul 02 18:33:06.577076 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.577058    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 02 18:33:06.577423 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.577398    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"} err="failed to get container status \"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14\": rpc error: code = NotFound desc = could not find container \"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14\": container with ID starting with e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14 not found: ID does not exist"
Jul 02 18:33:06.577506 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.577426    2575 scope.go:117] "RemoveContainer" containerID="263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"
Jul 02 18:33:06.577735 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.577707    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"} err="failed to get container status \"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152\": rpc error: code = NotFound desc = could not find container \"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152\": container with ID starting with 263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152 not found: ID does not exist"
Jul 02 18:33:06.577833 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.577736    2575 scope.go:117] "RemoveContainer" containerID="73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"
Jul 02 18:33:06.578277 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.578247    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"} err="failed to get container status \"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde\": rpc error: code = NotFound desc = could not find container \"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde\": container with ID starting with 73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde not found: ID does not exist"
Jul 02 18:33:06.578398 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.578278    2575 scope.go:117] "RemoveContainer" containerID="a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"
Jul 02 18:33:06.579924 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.579899    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 02 18:33:06.580285 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.580261    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"} err="failed to get container status \"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da\": rpc error: code = NotFound desc = could not find container \"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da\": container with ID starting with a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da not found: ID does not exist"
Jul 02 18:33:06.580369 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.580287    2575 scope.go:117] "RemoveContainer" containerID="e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"
Jul 02 18:33:06.580589 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.580560    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"} err="failed to get container status \"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0\": rpc error: code = NotFound desc = could not find container \"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0\": container with ID starting with e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0 not found: ID does not exist"
Jul 02 18:33:06.580647 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.580591    2575 scope.go:117] "RemoveContainer" containerID="1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"
Jul 02 18:33:06.581071 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.581045    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"} err="failed to get container status \"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14\": rpc error: code = NotFound desc = could not find container \"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14\": container with ID starting with 1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14 not found: ID does not exist"
Jul 02 18:33:06.581158 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.581095    2575 scope.go:117] "RemoveContainer" containerID="3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"
Jul 02 18:33:06.581659 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.581615    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"} err="failed to get container status \"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf\": rpc error: code = NotFound desc = could not find container \"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf\": container with ID starting with 3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf not found: ID does not exist"
Jul 02 18:33:06.581659 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.581641    2575 scope.go:117] "RemoveContainer" containerID="e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"
Jul 02 18:33:06.581892 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.581874    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"} err="failed to get container status \"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14\": rpc error: code = NotFound desc = could not find container \"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14\": container with ID starting with e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14 not found: ID does not exist"
Jul 02 18:33:06.581945 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.581901    2575 scope.go:117] "RemoveContainer" containerID="263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"
Jul 02 18:33:06.582162 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.582142    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"} err="failed to get container status \"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152\": rpc error: code = NotFound desc = could not find container \"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152\": container with ID starting with 263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152 not found: ID does not exist"
Jul 02 18:33:06.582162 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.582162    2575 scope.go:117] "RemoveContainer" containerID="73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"
Jul 02 18:33:06.582427 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.582409    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"} err="failed to get container status \"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde\": rpc error: code = NotFound desc = could not find container \"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde\": container with ID starting with 73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde not found: ID does not exist"
Jul 02 18:33:06.582500 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.582429    2575 scope.go:117] "RemoveContainer" containerID="a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"
Jul 02 18:33:06.582654 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.582636    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"} err="failed to get container status \"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da\": rpc error: code = NotFound desc = could not find container \"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da\": container with ID starting with a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da not found: ID does not exist"
Jul 02 18:33:06.582703 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.582655    2575 scope.go:117] "RemoveContainer" containerID="e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"
Jul 02 18:33:06.582878 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.582859    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"} err="failed to get container status \"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0\": rpc error: code = NotFound desc = could not find container \"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0\": container with ID starting with e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0 not found: ID does not exist"
Jul 02 18:33:06.582934 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.582880    2575 scope.go:117] "RemoveContainer" containerID="1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"
Jul 02 18:33:06.583114 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.583095    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"} err="failed to get container status \"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14\": rpc error: code = NotFound desc = could not find container \"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14\": container with ID starting with 1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14 not found: ID does not exist"
Jul 02 18:33:06.583114 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.583114    2575 scope.go:117] "RemoveContainer" containerID="3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"
Jul 02 18:33:06.583381 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.583362    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"} err="failed to get container status \"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf\": rpc error: code = NotFound desc = could not find container \"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf\": container with ID starting with 3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf not found: ID does not exist"
Jul 02 18:33:06.583453 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.583382    2575 scope.go:117] "RemoveContainer" containerID="e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"
Jul 02 18:33:06.583695 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.583671    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"} err="failed to get container status \"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14\": rpc error: code = NotFound desc = could not find container \"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14\": container with ID starting with e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14 not found: ID does not exist"
Jul 02 18:33:06.583802 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.583696    2575 scope.go:117] "RemoveContainer" containerID="263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"
Jul 02 18:33:06.584058 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.584036    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"} err="failed to get container status \"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152\": rpc error: code = NotFound desc = could not find container \"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152\": container with ID starting with 263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152 not found: ID does not exist"
Jul 02 18:33:06.584138 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.584059    2575 scope.go:117] "RemoveContainer" containerID="73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"
Jul 02 18:33:06.584492 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.584469    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"} err="failed to get container status \"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde\": rpc error: code = NotFound desc = could not find container \"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde\": container with ID starting with 73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde not found: ID does not exist"
Jul 02 18:33:06.584764 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.584495    2575 scope.go:117] "RemoveContainer" containerID="a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"
Jul 02 18:33:06.584838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.584819    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"} err="failed to get container status \"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da\": rpc error: code = NotFound desc = could not find container \"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da\": container with ID starting with a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da not found: ID does not exist"
Jul 02 18:33:06.584902 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.584842    2575 scope.go:117] "RemoveContainer" containerID="e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"
Jul 02 18:33:06.585100 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.585077    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"} err="failed to get container status \"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0\": rpc error: code = NotFound desc = could not find container \"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0\": container with ID starting with e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0 not found: ID does not exist"
Jul 02 18:33:06.585100 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.585099    2575 scope.go:117] "RemoveContainer" containerID="1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"
Jul 02 18:33:06.585395 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.585377    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"} err="failed to get container status \"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14\": rpc error: code = NotFound desc = could not find container \"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14\": container with ID starting with 1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14 not found: ID does not exist"
Jul 02 18:33:06.585454 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.585397    2575 scope.go:117] "RemoveContainer" containerID="3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"
Jul 02 18:33:06.585619 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.585601    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"} err="failed to get container status \"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf\": rpc error: code = NotFound desc = could not find container \"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf\": container with ID starting with 3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf not found: ID does not exist"
Jul 02 18:33:06.585682 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.585621    2575 scope.go:117] "RemoveContainer" containerID="e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"
Jul 02 18:33:06.585897 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.585880    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"} err="failed to get container status \"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14\": rpc error: code = NotFound desc = could not find container \"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14\": container with ID starting with e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14 not found: ID does not exist"
Jul 02 18:33:06.585968 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.585898    2575 scope.go:117] "RemoveContainer" containerID="263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"
Jul 02 18:33:06.586081 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.586062    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 18:33:06.586148 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.586130    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"} err="failed to get container status \"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152\": rpc error: code = NotFound desc = could not find container \"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152\": container with ID starting with 263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152 not found: ID does not exist"
Jul 02 18:33:06.586187 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.586150    2575 scope.go:117] "RemoveContainer" containerID="73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"
Jul 02 18:33:06.586410 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.586391    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"} err="failed to get container status \"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde\": rpc error: code = NotFound desc = could not find container \"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde\": container with ID starting with 73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde not found: ID does not exist"
Jul 02 18:33:06.586482 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.586412    2575 scope.go:117] "RemoveContainer" containerID="a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"
Jul 02 18:33:06.586632 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.586613    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"} err="failed to get container status \"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da\": rpc error: code = NotFound desc = could not find container \"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da\": container with ID starting with a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da not found: ID does not exist"
Jul 02 18:33:06.586673 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.586635    2575 scope.go:117] "RemoveContainer" containerID="e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"
Jul 02 18:33:06.586832 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.586809    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"} err="failed to get container status \"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0\": rpc error: code = NotFound desc = could not find container \"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0\": container with ID starting with e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0 not found: ID does not exist"
Jul 02 18:33:06.586873 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.586836    2575 scope.go:117] "RemoveContainer" containerID="1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"
Jul 02 18:33:06.587078 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.587061    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14"} err="failed to get container status \"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14\": rpc error: code = NotFound desc = could not find container \"1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14\": container with ID starting with 1fd821403ccf9c040a7eb4e117a6c3847a106703ac96494d87d72aae68f55a14 not found: ID does not exist"
Jul 02 18:33:06.587126 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.587079    2575 scope.go:117] "RemoveContainer" containerID="3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"
Jul 02 18:33:06.587336 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.587316    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf"} err="failed to get container status \"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf\": rpc error: code = NotFound desc = could not find container \"3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf\": container with ID starting with 3582806502b253206d86bf1a3bba5d4716e731712ae89f06e9b8e96a3f67bdbf not found: ID does not exist"
Jul 02 18:33:06.587336 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.587337    2575 scope.go:117] "RemoveContainer" containerID="e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"
Jul 02 18:33:06.587610 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.587590    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14"} err="failed to get container status \"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14\": rpc error: code = NotFound desc = could not find container \"e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14\": container with ID starting with e22b83fffb8b7133ef6ac9bf7d2b60d6bcf2cc693c3643fcd0b1d0d32c6e6c14 not found: ID does not exist"
Jul 02 18:33:06.587684 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.587611    2575 scope.go:117] "RemoveContainer" containerID="263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"
Jul 02 18:33:06.587869 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.587849    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152"} err="failed to get container status \"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152\": rpc error: code = NotFound desc = could not find container \"263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152\": container with ID starting with 263ea4603837f7b688314fb999ac4f2cc3ae9fe10f2f80914e9b53c66b2bb152 not found: ID does not exist"
Jul 02 18:33:06.587926 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.587870    2575 scope.go:117] "RemoveContainer" containerID="73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"
Jul 02 18:33:06.588100 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.588082    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde"} err="failed to get container status \"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde\": rpc error: code = NotFound desc = could not find container \"73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde\": container with ID starting with 73d863e0e3950e277300867ba60b51ba257a6527fac93042a9d57b4562b51cde not found: ID does not exist"
Jul 02 18:33:06.588147 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.588101    2575 scope.go:117] "RemoveContainer" containerID="a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"
Jul 02 18:33:06.588360 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.588342    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da"} err="failed to get container status \"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da\": rpc error: code = NotFound desc = could not find container \"a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da\": container with ID starting with a371edaba9530fca66da285dc97a9cea8b2e01511f81ba9aa08da5317011b5da not found: ID does not exist"
Jul 02 18:33:06.588420 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.588362    2575 scope.go:117] "RemoveContainer" containerID="e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"
Jul 02 18:33:06.588587 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.588569    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0"} err="failed to get container status \"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0\": rpc error: code = NotFound desc = could not find container \"e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0\": container with ID starting with e8ab18e9edc8ff85c444cc4a63cd8adc10bae35d9636f8505da36f8f01431bd0 not found: ID does not exist"
Jul 02 18:33:06.608019 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.607990    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608141 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608141 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608141 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608066    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608281 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608128    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608281 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608168    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-config-out\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608281 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608218    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-web-config\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608281 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608251    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608406 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608288    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608406 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608322    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608406 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608342    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608406 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608402    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608520 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608498    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-config\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608549 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608518    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608549 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608534    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608610 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608555    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608610 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608672 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608624    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608672 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608641    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.608672 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.608662    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpnwr\" (UniqueName: \"kubernetes.io/projected/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-kube-api-access-mpnwr\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.709409 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.709368    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.709409 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.709416    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.709648 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.709436    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-config\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.709648 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.709454    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.709648 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.709470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.709648 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.709488    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.709648 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.709507    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.709926 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.709900    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.709966 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.709947    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.710000 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.709979    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpnwr\" (UniqueName: \"kubernetes.io/projected/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-kube-api-access-mpnwr\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.710042 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.710027    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.710077 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.710059    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.710107 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.710084    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.710141 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.710108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.710174 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.710157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.710254 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.710186    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-config-out\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.710254 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.710245    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-web-config\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.710319 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.710270    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.710354 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.710317    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.710386 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.710349    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.710559 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.710539    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.710869 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.710850    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.712018 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.711126    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.712018 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.711514    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.712018 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.711719    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.712453 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.712424    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.712799 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.712769    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.713084 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.713058    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.713805 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.713760    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.713805 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.713777    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.713958 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.713823    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-config\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.714621 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.714593    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.714786 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.714743    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.714786 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.714773    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.714932 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.714865    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.714932 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.714889    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.715220 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.715165    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.715754 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.715737    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-web-config\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.716117 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.716099    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-config-out\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.719917 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.719898    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpnwr\" (UniqueName: \"kubernetes.io/projected/b19f68e7-3fe0-42c8-9487-e90ac8e68e26-kube-api-access-mpnwr\") pod \"prometheus-k8s-0\" (UID: \"b19f68e7-3fe0-42c8-9487-e90ac8e68e26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:06.881799 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:06.881724    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:07.012541 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:07.012516    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 18:33:07.519745 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:07.519708    2575 generic.go:356] "Generic (PLEG): container finished" podID="b19f68e7-3fe0-42c8-9487-e90ac8e68e26" containerID="0b6f9d9cdb6887b9dea452e6b20f6b470b7d9b9c4e1016cb3c3b201cdd6e2ff7" exitCode=0
Jul 02 18:33:07.520211 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:07.519798    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b19f68e7-3fe0-42c8-9487-e90ac8e68e26","Type":"ContainerDied","Data":"0b6f9d9cdb6887b9dea452e6b20f6b470b7d9b9c4e1016cb3c3b201cdd6e2ff7"}
Jul 02 18:33:07.520211 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:07.519879    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b19f68e7-3fe0-42c8-9487-e90ac8e68e26","Type":"ContainerStarted","Data":"e81040eeaf734a1236cbd9b548edeacb1246065e906f617ec03fc90d35a3b5f3"}
Jul 02 18:33:08.489900 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:08.489868    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01f465e2-fe9b-49ce-a4cc-849e78e19611" path="/var/lib/kubelet/pods/01f465e2-fe9b-49ce-a4cc-849e78e19611/volumes"
Jul 02 18:33:08.528040 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:08.528007    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b19f68e7-3fe0-42c8-9487-e90ac8e68e26","Type":"ContainerStarted","Data":"6ea9b79337223870ff256c5b3973c143a3ecad7a961e546c61a54b8cc9dba0f8"}
Jul 02 18:33:08.528439 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:08.528076    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b19f68e7-3fe0-42c8-9487-e90ac8e68e26","Type":"ContainerStarted","Data":"84c56061a233d942ab05a0b2436f40348dacd4914b2318d733d2b8cbd37423a7"}
Jul 02 18:33:08.528439 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:08.528102    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b19f68e7-3fe0-42c8-9487-e90ac8e68e26","Type":"ContainerStarted","Data":"2e96a3d4c51c7593037df37e085e694fc16e67ca7e9922d04a58b54524e8e2a1"}
Jul 02 18:33:08.528439 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:08.528126    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b19f68e7-3fe0-42c8-9487-e90ac8e68e26","Type":"ContainerStarted","Data":"4b608be069a39930bed3f5337520354b224b4331d6fc6ff0fe6f61444580440b"}
Jul 02 18:33:08.528439 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:08.528151    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b19f68e7-3fe0-42c8-9487-e90ac8e68e26","Type":"ContainerStarted","Data":"5fef8b378f6cb5a03f94a2d74b95735376a633edf763d7fc94bac4a0a67eaa9c"}
Jul 02 18:33:08.528439 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:08.528182    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b19f68e7-3fe0-42c8-9487-e90ac8e68e26","Type":"ContainerStarted","Data":"514454f89fda78bd689c21269907e5cb150b657cd55ca9daa3f931299fda813b"}
Jul 02 18:33:08.557312 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:08.557248    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.5328823700000003 podStartE2EDuration="2.557228183s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.345813ms" isStatefulPod=false podCreationTimestamp="2026-07-02 18:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:33:08.554603952 +0000 UTC m=+180.624231512" watchObservedRunningTime="2026-07-02 18:33:08.557228183 +0000 UTC m=+180.626855714"
Jul 02 18:33:09.401617 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:09.401578    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:09.453040 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:09.453001    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:11.882852 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:11.882818    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:33:12.377570 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:12.377525    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:12.476620 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:12.476587    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:12.577041 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:12.577004    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:12.784719 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:12.784632    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:12.883816 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:12.883787    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:13.156497 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:13.156464    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:13.211561 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:13.211527    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:14.037760 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:14.037729    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:14.131640 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:14.131603    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:23.259457 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:23.259420    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:23.310299 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:23.310262    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:23.367021 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:23.366991    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:28.958002 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:28.957958    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:29.055872 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:29.055832    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:29.120741 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:29.120706    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:32.184248 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:32.184211    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:32.236294 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:32.236241    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:33:56.061607 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:33:56.061573    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:34:00.347386 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:34:00.347340    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:34:06.881933 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:34:06.881889    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:34:06.897406 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:34:06.897377    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:34:07.765151 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:34:07.765126    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 18:34:38.974687 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:34:38.974603    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:34:39.027340 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:34:39.027301    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:34:39.314019 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:34:39.313984    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:34:39.413715 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:34:39.413678    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:34:39.513533 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:34:39.513495    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:00.568977 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:00.568937    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:01.151792 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:01.151751    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:01.249341 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:01.249301    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:07.563756 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:07.563714    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:07.736118 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:07.736081    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:07.836668 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:07.836572    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:08.034363 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:08.034326    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:08.328627 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:08.328599    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 18:35:08.334862 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:08.334832    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 18:35:08.337120 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:08.337097    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 18:35:08.340327 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:08.340302    2575 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 18:35:08.344037 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:08.344017    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 18:35:08.481329 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:08.481159    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:08.623337 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:08.623251    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:08.678300 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:08.678271    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:09.006000 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:09.005905    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:09.104688 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:09.104653    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:09.204432 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:09.204392    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:09.425810 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:09.425776    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:09.511860 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:09.511819    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:53.245829 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:53.245787    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:35:54.704024 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:35:54.703981    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:36:13.745646 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:13.745552    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:36:13.820943 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:13.820894    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:36:13.851323 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:13.851296    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-j5hhk"]
Jul 02 18:36:13.853352 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:13.853336    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-j5hhk"
Jul 02 18:36:13.856254 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:13.856231    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 18:36:13.856386 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:13.856259    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 18:36:13.856386 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:13.856259    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 18:36:13.856386 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:13.856363    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-fwghs\""
Jul 02 18:36:13.863181 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:13.863151    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-j5hhk"]
Jul 02 18:36:13.889477 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:13.889451    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:36:13.917944 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:13.917908    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzr7\" (UniqueName: \"kubernetes.io/projected/e6fd7045-82f3-408e-95c0-9b5445775044-kube-api-access-6jzr7\") pod \"seaweedfs-64568bcd49-j5hhk\" (UID: \"e6fd7045-82f3-408e-95c0-9b5445775044\") " pod="kserve/seaweedfs-64568bcd49-j5hhk"
Jul 02 18:36:13.918098 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:13.917948    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e6fd7045-82f3-408e-95c0-9b5445775044-data\") pod \"seaweedfs-64568bcd49-j5hhk\" (UID: \"e6fd7045-82f3-408e-95c0-9b5445775044\") " pod="kserve/seaweedfs-64568bcd49-j5hhk"
Jul 02 18:36:13.945457 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:13.945429    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:36:14.018688 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:14.018594    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzr7\" (UniqueName: \"kubernetes.io/projected/e6fd7045-82f3-408e-95c0-9b5445775044-kube-api-access-6jzr7\") pod \"seaweedfs-64568bcd49-j5hhk\" (UID: \"e6fd7045-82f3-408e-95c0-9b5445775044\") " pod="kserve/seaweedfs-64568bcd49-j5hhk"
Jul 02 18:36:14.018688 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:14.018639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e6fd7045-82f3-408e-95c0-9b5445775044-data\") pod \"seaweedfs-64568bcd49-j5hhk\" (UID: \"e6fd7045-82f3-408e-95c0-9b5445775044\") " pod="kserve/seaweedfs-64568bcd49-j5hhk"
Jul 02 18:36:14.018991 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:14.018975    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e6fd7045-82f3-408e-95c0-9b5445775044-data\") pod \"seaweedfs-64568bcd49-j5hhk\" (UID: \"e6fd7045-82f3-408e-95c0-9b5445775044\") " pod="kserve/seaweedfs-64568bcd49-j5hhk"
Jul 02 18:36:14.027409 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:14.027388    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzr7\" (UniqueName: \"kubernetes.io/projected/e6fd7045-82f3-408e-95c0-9b5445775044-kube-api-access-6jzr7\") pod \"seaweedfs-64568bcd49-j5hhk\" (UID: \"e6fd7045-82f3-408e-95c0-9b5445775044\") " pod="kserve/seaweedfs-64568bcd49-j5hhk"
Jul 02 18:36:14.076055 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:14.076011    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:36:14.161948 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:14.161918    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-j5hhk"
Jul 02 18:36:14.286783 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:14.286761    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-j5hhk"]
Jul 02 18:36:14.288781 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:36:14.288756    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fd7045_82f3_408e_95c0_9b5445775044.slice/crio-2c7618783b636aa65d5f6153f22beee9c8b888a0b00f2128a2544c2743aea3dc WatchSource:0}: Error finding container 2c7618783b636aa65d5f6153f22beee9c8b888a0b00f2128a2544c2743aea3dc: Status 404 returned error can't find the container with id 2c7618783b636aa65d5f6153f22beee9c8b888a0b00f2128a2544c2743aea3dc
Jul 02 18:36:14.290161 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:14.290141    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 18:36:14.570153 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:14.570122    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:36:14.668961 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:14.668931    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:36:14.803867 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:14.803831    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:36:15.217473 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:15.217438    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-j5hhk" event={"ID":"e6fd7045-82f3-408e-95c0-9b5445775044","Type":"ContainerStarted","Data":"2c7618783b636aa65d5f6153f22beee9c8b888a0b00f2128a2544c2743aea3dc"}
Jul 02 18:36:17.180013 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:17.179976    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:36:17.298542 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:17.298507    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:36:18.230100 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:18.230063    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-j5hhk" event={"ID":"e6fd7045-82f3-408e-95c0-9b5445775044","Type":"ContainerStarted","Data":"8e1fc915b94fa438b2f46731cbdb826e36b33863a54dbcdda1d81472b1300a87"}
Jul 02 18:36:18.230543 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:18.230217    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-j5hhk"
Jul 02 18:36:18.247445 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:18.247397    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-j5hhk" podStartSLOduration=1.446171225 podStartE2EDuration="5.247380034s" totalImagesPullingTime="3.801208809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:36:18.245944426 +0000 UTC m=+370.315571957" watchObservedRunningTime="2026-07-02 18:36:18.247380034 +0000 UTC m=+370.317007562"
Jul 02 18:36:24.236080 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:36:24.236045    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-j5hhk"
Jul 02 18:37:12.962032 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:12.961995    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:37:23.128540 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:23.128480    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:37:23.205646 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:23.205605    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:37:23.423023 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:23.422927    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:37:23.613753 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:23.613719    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:37:23.814888 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:23.814853    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:37:23.915908 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:23.915871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:37:24.303773 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:24.303737    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:37:39.029467 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:39.029386    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-458lw"]
Jul 02 18:37:39.032637 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:39.032619    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-458lw"
Jul 02 18:37:39.039619 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:39.039454    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-458lw"]
Jul 02 18:37:39.125037 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:39.125003    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c37bff-72fe-4e97-b475-8d6beb710aac-model-cache\") pod \"s3-init-458lw\" (UID: \"16c37bff-72fe-4e97-b475-8d6beb710aac\") " pod="kserve/s3-init-458lw"
Jul 02 18:37:39.125237 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:39.125049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvtv8\" (UniqueName: \"kubernetes.io/projected/16c37bff-72fe-4e97-b475-8d6beb710aac-kube-api-access-mvtv8\") pod \"s3-init-458lw\" (UID: \"16c37bff-72fe-4e97-b475-8d6beb710aac\") " pod="kserve/s3-init-458lw"
Jul 02 18:37:39.226547 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:39.226512    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvtv8\" (UniqueName: \"kubernetes.io/projected/16c37bff-72fe-4e97-b475-8d6beb710aac-kube-api-access-mvtv8\") pod \"s3-init-458lw\" (UID: \"16c37bff-72fe-4e97-b475-8d6beb710aac\") " pod="kserve/s3-init-458lw"
Jul 02 18:37:39.226736 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:39.226586    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c37bff-72fe-4e97-b475-8d6beb710aac-model-cache\") pod \"s3-init-458lw\" (UID: \"16c37bff-72fe-4e97-b475-8d6beb710aac\") " pod="kserve/s3-init-458lw"
Jul 02 18:37:39.226951 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:39.226931    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c37bff-72fe-4e97-b475-8d6beb710aac-model-cache\") pod \"s3-init-458lw\" (UID: \"16c37bff-72fe-4e97-b475-8d6beb710aac\") " pod="kserve/s3-init-458lw"
Jul 02 18:37:39.234911 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:39.234872    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvtv8\" (UniqueName: \"kubernetes.io/projected/16c37bff-72fe-4e97-b475-8d6beb710aac-kube-api-access-mvtv8\") pod \"s3-init-458lw\" (UID: \"16c37bff-72fe-4e97-b475-8d6beb710aac\") " pod="kserve/s3-init-458lw"
Jul 02 18:37:39.354889 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:39.354851    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-458lw"
Jul 02 18:37:39.481049 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:39.481024    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-458lw"]
Jul 02 18:37:39.482937 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:37:39.482910    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16c37bff_72fe_4e97_b475_8d6beb710aac.slice/crio-1681bac839c5a671ef06747a5df600837365df54c230e3aaa9ff295ceb37a28e WatchSource:0}: Error finding container 1681bac839c5a671ef06747a5df600837365df54c230e3aaa9ff295ceb37a28e: Status 404 returned error can't find the container with id 1681bac839c5a671ef06747a5df600837365df54c230e3aaa9ff295ceb37a28e
Jul 02 18:37:39.530626 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:39.530592    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-458lw" event={"ID":"16c37bff-72fe-4e97-b475-8d6beb710aac","Type":"ContainerStarted","Data":"1681bac839c5a671ef06747a5df600837365df54c230e3aaa9ff295ceb37a28e"}
Jul 02 18:37:44.551225 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:44.551175    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-458lw" event={"ID":"16c37bff-72fe-4e97-b475-8d6beb710aac","Type":"ContainerStarted","Data":"50b62a6b711862d3a550368f5acd48c95e6e9285e5b270e6528a44f350772d3c"}
Jul 02 18:37:46.561403 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:46.561374    2575 generic.go:356] "Generic (PLEG): container finished" podID="16c37bff-72fe-4e97-b475-8d6beb710aac" containerID="50b62a6b711862d3a550368f5acd48c95e6e9285e5b270e6528a44f350772d3c" exitCode=0
Jul 02 18:37:46.561751 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:46.561411    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-458lw" event={"ID":"16c37bff-72fe-4e97-b475-8d6beb710aac","Type":"ContainerDied","Data":"50b62a6b711862d3a550368f5acd48c95e6e9285e5b270e6528a44f350772d3c"}
Jul 02 18:37:52.593705 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:52.593666    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-458lw" event={"ID":"16c37bff-72fe-4e97-b475-8d6beb710aac","Type":"ContainerStarted","Data":"cc3f2b8246d39479c5d25285070d9228abba15895750bf3ba6af923e8bc98d29"}
Jul 02 18:37:52.612221 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:37:52.612143    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-458lw" podStartSLOduration=1.710468782 podStartE2EDuration="13.612104916s" totalImagesPullingTime="9.622216161s" totalInitContainerRuntime="2.279419973s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:37:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:37:52.61091541 +0000 UTC m=+464.680542937" watchObservedRunningTime="2026-07-02 18:37:52.612104916 +0000 UTC m=+464.681732443"
Jul 02 18:38:56.845297 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:38:56.845261    2575 generic.go:356] "Generic (PLEG): container finished" podID="16c37bff-72fe-4e97-b475-8d6beb710aac" containerID="cc3f2b8246d39479c5d25285070d9228abba15895750bf3ba6af923e8bc98d29" exitCode=0
Jul 02 18:38:56.845703 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:38:56.845335    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-458lw" event={"ID":"16c37bff-72fe-4e97-b475-8d6beb710aac","Type":"ContainerDied","Data":"cc3f2b8246d39479c5d25285070d9228abba15895750bf3ba6af923e8bc98d29"}
Jul 02 18:38:57.967330 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:38:57.967303    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-458lw"
Jul 02 18:38:58.167015 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:38:58.166934    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c37bff-72fe-4e97-b475-8d6beb710aac-model-cache\") pod \"16c37bff-72fe-4e97-b475-8d6beb710aac\" (UID: \"16c37bff-72fe-4e97-b475-8d6beb710aac\") "
Jul 02 18:38:58.167015 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:38:58.166968    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvtv8\" (UniqueName: \"kubernetes.io/projected/16c37bff-72fe-4e97-b475-8d6beb710aac-kube-api-access-mvtv8\") pod \"16c37bff-72fe-4e97-b475-8d6beb710aac\" (UID: \"16c37bff-72fe-4e97-b475-8d6beb710aac\") "
Jul 02 18:38:58.169160 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:38:58.169136    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c37bff-72fe-4e97-b475-8d6beb710aac-kube-api-access-mvtv8" (OuterVolumeSpecName: "kube-api-access-mvtv8") pod "16c37bff-72fe-4e97-b475-8d6beb710aac" (UID: "16c37bff-72fe-4e97-b475-8d6beb710aac"). InnerVolumeSpecName "kube-api-access-mvtv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:38:58.185431 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:38:58.185403    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c37bff-72fe-4e97-b475-8d6beb710aac-model-cache" (OuterVolumeSpecName: "model-cache") pod "16c37bff-72fe-4e97-b475-8d6beb710aac" (UID: "16c37bff-72fe-4e97-b475-8d6beb710aac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:38:58.267929 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:38:58.267893    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c37bff-72fe-4e97-b475-8d6beb710aac-model-cache\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:38:58.267929 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:38:58.267923    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvtv8\" (UniqueName: \"kubernetes.io/projected/16c37bff-72fe-4e97-b475-8d6beb710aac-kube-api-access-mvtv8\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:38:58.854606 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:38:58.854575    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-458lw"
Jul 02 18:38:58.854606 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:38:58.854594    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-458lw" event={"ID":"16c37bff-72fe-4e97-b475-8d6beb710aac","Type":"ContainerDied","Data":"1681bac839c5a671ef06747a5df600837365df54c230e3aaa9ff295ceb37a28e"}
Jul 02 18:38:58.854809 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:38:58.854625    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1681bac839c5a671ef06747a5df600837365df54c230e3aaa9ff295ceb37a28e"
Jul 02 18:38:59.253171 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:38:59.253077    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:38:59.302147 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:38:59.302112    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:39:11.466476 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.466372    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:39:11.566532 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.566490    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:39:11.755728 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.755649    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:39:11.789660 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.789628    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"]
Jul 02 18:39:11.789904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.789892    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c37bff-72fe-4e97-b475-8d6beb710aac" containerName="download-hf-model"
Jul 02 18:39:11.789951 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.789905    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c37bff-72fe-4e97-b475-8d6beb710aac" containerName="download-hf-model"
Jul 02 18:39:11.789951 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.789926    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c37bff-72fe-4e97-b475-8d6beb710aac" containerName="s3-init"
Jul 02 18:39:11.789951 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.789932    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c37bff-72fe-4e97-b475-8d6beb710aac" containerName="s3-init"
Jul 02 18:39:11.790041 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.789992    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c37bff-72fe-4e97-b475-8d6beb710aac" containerName="s3-init"
Jul 02 18:39:11.792891 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.792877    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.795690 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.795670    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 02 18:39:11.795838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.795734    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 18:39:11.795838 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.795828    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-t4t2c\""
Jul 02 18:39:11.795960 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.795832    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 02 18:39:11.796057 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.796041    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 18:39:11.797143 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.797121    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 18:39:11.797265 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.797125    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 18:39:11.803677 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.803655    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"]
Jul 02 18:39:11.855239 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.855187    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:39:11.876895 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.876867    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.876895 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.876896    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ckc7\" (UniqueName: \"kubernetes.io/projected/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kube-api-access-7ckc7\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.877072 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.876919    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.877072 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.876957    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.877072 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.877040    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.881559 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.881536    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"]
Jul 02 18:39:11.885029 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.885010    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:39:11.887913 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.887895    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bb242-kube-rbac-proxy-sar-config\""
Jul 02 18:39:11.887995 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.887936    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bb242-predictor-serving-cert\""
Jul 02 18:39:11.895461 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.895439    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"]
Jul 02 18:39:11.957090 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.957059    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:39:11.978374 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.978350    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.978522 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.978394    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e792528-8683-400b-a079-aa659d1a1c5f-proxy-tls\") pod \"error-404-isvc-bb242-predictor-7d85f5f688-cxpx6\" (UID: \"5e792528-8683-400b-a079-aa659d1a1c5f\") " pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:39:11.978522 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.978413    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e792528-8683-400b-a079-aa659d1a1c5f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bb242-predictor-7d85f5f688-cxpx6\" (UID: \"5e792528-8683-400b-a079-aa659d1a1c5f\") " pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:39:11.978522 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.978502    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh68f\" (UniqueName: \"kubernetes.io/projected/5e792528-8683-400b-a079-aa659d1a1c5f-kube-api-access-dh68f\") pod \"error-404-isvc-bb242-predictor-7d85f5f688-cxpx6\" (UID: \"5e792528-8683-400b-a079-aa659d1a1c5f\") " pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:39:11.978640 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.978534    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.978640 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.978567    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.978640 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.978621    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ckc7\" (UniqueName: \"kubernetes.io/projected/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kube-api-access-7ckc7\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.978761 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.978661    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.978761 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.978719    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.979296 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.979275    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.979408 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.979277    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.980893 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.980874    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.990455 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.990432    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ckc7\" (UniqueName: \"kubernetes.io/projected/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kube-api-access-7ckc7\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:11.993872 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.993851    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"]
Jul 02 18:39:11.997467 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:11.997448    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.000692 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.000673    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 02 18:39:12.000791 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.000713    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 02 18:39:12.012872 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.012798    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"]
Jul 02 18:39:12.042303 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.041229    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:39:12.079706 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.079667    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e792528-8683-400b-a079-aa659d1a1c5f-proxy-tls\") pod \"error-404-isvc-bb242-predictor-7d85f5f688-cxpx6\" (UID: \"5e792528-8683-400b-a079-aa659d1a1c5f\") " pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:39:12.079706 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.079705    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e792528-8683-400b-a079-aa659d1a1c5f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bb242-predictor-7d85f5f688-cxpx6\" (UID: \"5e792528-8683-400b-a079-aa659d1a1c5f\") " pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:39:12.079950 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.079727    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh68f\" (UniqueName: \"kubernetes.io/projected/5e792528-8683-400b-a079-aa659d1a1c5f-kube-api-access-dh68f\") pod \"error-404-isvc-bb242-predictor-7d85f5f688-cxpx6\" (UID: \"5e792528-8683-400b-a079-aa659d1a1c5f\") " pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:39:12.079950 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.079803    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.079950 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.079834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.079950 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.079850    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plcsr\" (UniqueName: \"kubernetes.io/projected/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kube-api-access-plcsr\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.079950 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.079871    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.079950 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.079896    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.080485 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.080458    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e792528-8683-400b-a079-aa659d1a1c5f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bb242-predictor-7d85f5f688-cxpx6\" (UID: \"5e792528-8683-400b-a079-aa659d1a1c5f\") " pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:39:12.082456 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.082427    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e792528-8683-400b-a079-aa659d1a1c5f-proxy-tls\") pod \"error-404-isvc-bb242-predictor-7d85f5f688-cxpx6\" (UID: \"5e792528-8683-400b-a079-aa659d1a1c5f\") " pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:39:12.088898 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.088868    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh68f\" (UniqueName: \"kubernetes.io/projected/5e792528-8683-400b-a079-aa659d1a1c5f-kube-api-access-dh68f\") pod \"error-404-isvc-bb242-predictor-7d85f5f688-cxpx6\" (UID: \"5e792528-8683-400b-a079-aa659d1a1c5f\") " pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:39:12.093813 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.093788    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"]
Jul 02 18:39:12.098010 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.097990    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.100796 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.100775    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 02 18:39:12.100905 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.100845    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 02 18:39:12.103286 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.103253    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:12.105023 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.104994    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:39:12.107094 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.107022    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"]
Jul 02 18:39:12.181187 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.181159    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b12d2ba-3425-4dc3-b91d-e709f8436b48-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.181342 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.181230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b12d2ba-3425-4dc3-b91d-e709f8436b48-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.181342 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.181257    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4c5c\" (UniqueName: \"kubernetes.io/projected/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kube-api-access-z4c5c\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.181342 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.181277    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.181469 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.181383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.181469 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.181432    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.181560 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.181468    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.181560 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.181493    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plcsr\" (UniqueName: \"kubernetes.io/projected/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kube-api-access-plcsr\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.181659 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.181560    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.181659 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.181597    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.182037 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.182012    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.182369 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.182350    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.182369 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.182360    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.183990 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.183947    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.189507 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.189485    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plcsr\" (UniqueName: \"kubernetes.io/projected/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kube-api-access-plcsr\") pod \"isvc-xgboost-graph-predictor-555fc69d56-c5zrq\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.196317 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.196294    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:39:12.201103 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.201051    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:39:12.239714 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.239684    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"]
Jul 02 18:39:12.240376 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:39:12.240344    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43aa4427_6e7b_40ba_9e27_6a4fa1d3fe97.slice/crio-7ea26104425f2ea9190db5f803a22029487122008364c40c2b650ecd4e239332 WatchSource:0}: Error finding container 7ea26104425f2ea9190db5f803a22029487122008364c40c2b650ecd4e239332: Status 404 returned error can't find the container with id 7ea26104425f2ea9190db5f803a22029487122008364c40c2b650ecd4e239332
Jul 02 18:39:12.285118 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.285088    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4c5c\" (UniqueName: \"kubernetes.io/projected/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kube-api-access-z4c5c\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.285263 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.285130    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.285263 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.285174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.285263 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.285238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b12d2ba-3425-4dc3-b91d-e709f8436b48-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.285430 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.285289    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b12d2ba-3425-4dc3-b91d-e709f8436b48-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.285430 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:39:12.285403    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 02 18:39:12.285532 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:39:12.285463    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b12d2ba-3425-4dc3-b91d-e709f8436b48-proxy-tls podName:1b12d2ba-3425-4dc3-b91d-e709f8436b48 nodeName:}" failed. No retries permitted until 2026-07-02 18:39:12.78544271 +0000 UTC m=+544.855070221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1b12d2ba-3425-4dc3-b91d-e709f8436b48-proxy-tls") pod "isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" (UID: "1b12d2ba-3425-4dc3-b91d-e709f8436b48") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 02 18:39:12.285610 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.285584    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.285970 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.285949    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.286057 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.285980    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b12d2ba-3425-4dc3-b91d-e709f8436b48-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.296132 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.296081    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4c5c\" (UniqueName: \"kubernetes.io/projected/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kube-api-access-z4c5c\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.306962 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.306921    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:12.337792 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.337699    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"]
Jul 02 18:39:12.339912 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:39:12.339876    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e792528_8683_400b_a079_aa659d1a1c5f.slice/crio-6f29967eb7df0227a882b4f6d9e0ece3898bafeac6672353ba7d0d591e658f7c WatchSource:0}: Error finding container 6f29967eb7df0227a882b4f6d9e0ece3898bafeac6672353ba7d0d591e658f7c: Status 404 returned error can't find the container with id 6f29967eb7df0227a882b4f6d9e0ece3898bafeac6672353ba7d0d591e658f7c
Jul 02 18:39:12.381094 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.381064    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:39:12.440314 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.440290    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"]
Jul 02 18:39:12.442697 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:39:12.442674    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda245f3bd_e1b2_4adb_a2d2_f38e34b31858.slice/crio-e8775149fef048802f0d4bd4a9d91e4b4ba19f17a836d6cc8eef0f2b77036659 WatchSource:0}: Error finding container e8775149fef048802f0d4bd4a9d91e4b4ba19f17a836d6cc8eef0f2b77036659: Status 404 returned error can't find the container with id e8775149fef048802f0d4bd4a9d91e4b4ba19f17a836d6cc8eef0f2b77036659
Jul 02 18:39:12.789581 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.789541    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b12d2ba-3425-4dc3-b91d-e709f8436b48-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.791965 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.791944    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b12d2ba-3425-4dc3-b91d-e709f8436b48-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:12.913236 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.913180    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" event={"ID":"a245f3bd-e1b2-4adb-a2d2-f38e34b31858","Type":"ContainerStarted","Data":"b1f34d84a5d2d9a91241e3b45ed66a0a25f645b905e9bff1f862b0a469896c9b"}
Jul 02 18:39:12.913428 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.913274    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" event={"ID":"a245f3bd-e1b2-4adb-a2d2-f38e34b31858","Type":"ContainerStarted","Data":"e8775149fef048802f0d4bd4a9d91e4b4ba19f17a836d6cc8eef0f2b77036659"}
Jul 02 18:39:12.916663 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.916626    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" event={"ID":"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97","Type":"ContainerStarted","Data":"c3084f09f4d5dbdba1bd75e469e7d52706db52e26cc8613b1d19014fc282bd9b"}
Jul 02 18:39:12.916810 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.916702    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" event={"ID":"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97","Type":"ContainerStarted","Data":"7ea26104425f2ea9190db5f803a22029487122008364c40c2b650ecd4e239332"}
Jul 02 18:39:12.920060 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:12.920029    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" event={"ID":"5e792528-8683-400b-a079-aa659d1a1c5f","Type":"ContainerStarted","Data":"6f29967eb7df0227a882b4f6d9e0ece3898bafeac6672353ba7d0d591e658f7c"}
Jul 02 18:39:13.010501 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:13.010451    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:13.226513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:13.226478    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"]
Jul 02 18:39:13.229014 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:39:13.228984    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b12d2ba_3425_4dc3_b91d_e709f8436b48.slice/crio-eac51bddeea7eb3f01b5c3278e5fc48bca94c0a8115c1c6f2ce8482692017c7a WatchSource:0}: Error finding container eac51bddeea7eb3f01b5c3278e5fc48bca94c0a8115c1c6f2ce8482692017c7a: Status 404 returned error can't find the container with id eac51bddeea7eb3f01b5c3278e5fc48bca94c0a8115c1c6f2ce8482692017c7a
Jul 02 18:39:13.929602 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:13.929560    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" event={"ID":"1b12d2ba-3425-4dc3-b91d-e709f8436b48","Type":"ContainerStarted","Data":"dec416e76609529b7ad6f122c2ce8d9147cf468ef86bcd5c51847d685e8749f9"}
Jul 02 18:39:13.930254 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:13.929643    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" event={"ID":"1b12d2ba-3425-4dc3-b91d-e709f8436b48","Type":"ContainerStarted","Data":"eac51bddeea7eb3f01b5c3278e5fc48bca94c0a8115c1c6f2ce8482692017c7a"}
Jul 02 18:39:17.953177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:17.953126    2575 generic.go:356] "Generic (PLEG): container finished" podID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerID="b1f34d84a5d2d9a91241e3b45ed66a0a25f645b905e9bff1f862b0a469896c9b" exitCode=0
Jul 02 18:39:17.953644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:17.953219    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" event={"ID":"a245f3bd-e1b2-4adb-a2d2-f38e34b31858","Type":"ContainerDied","Data":"b1f34d84a5d2d9a91241e3b45ed66a0a25f645b905e9bff1f862b0a469896c9b"}
Jul 02 18:39:17.955249 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:17.955162    2575 generic.go:356] "Generic (PLEG): container finished" podID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerID="c3084f09f4d5dbdba1bd75e469e7d52706db52e26cc8613b1d19014fc282bd9b" exitCode=0
Jul 02 18:39:17.955249 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:17.955226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" event={"ID":"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97","Type":"ContainerDied","Data":"c3084f09f4d5dbdba1bd75e469e7d52706db52e26cc8613b1d19014fc282bd9b"}
Jul 02 18:39:17.956966 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:17.956941    2575 generic.go:356] "Generic (PLEG): container finished" podID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerID="dec416e76609529b7ad6f122c2ce8d9147cf468ef86bcd5c51847d685e8749f9" exitCode=0
Jul 02 18:39:17.957080 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:17.956968    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" event={"ID":"1b12d2ba-3425-4dc3-b91d-e709f8436b48","Type":"ContainerDied","Data":"dec416e76609529b7ad6f122c2ce8d9147cf468ef86bcd5c51847d685e8749f9"}
Jul 02 18:39:45.152419 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:45.152382    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" event={"ID":"5e792528-8683-400b-a079-aa659d1a1c5f","Type":"ContainerStarted","Data":"dfa22108b05967bf93c60f03fa39e3dbc2394a3e7419a3dd03b7f80c5c76fe03"}
Jul 02 18:39:45.157002 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:45.156941    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" event={"ID":"1b12d2ba-3425-4dc3-b91d-e709f8436b48","Type":"ContainerStarted","Data":"b8a770043c59ee4127446981377122c06bbd35e1401da14dbc4eef73abb23e48"}
Jul 02 18:39:45.162079 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:45.161892    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" event={"ID":"a245f3bd-e1b2-4adb-a2d2-f38e34b31858","Type":"ContainerStarted","Data":"0e8e45a366d8886a9489c7b62dc377d3b595fb19af7594cc27b05d9a6b527a1d"}
Jul 02 18:39:45.166523 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:45.166425    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" event={"ID":"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97","Type":"ContainerStarted","Data":"69bb3c286363b8b7222474e2f69ff7ddad4d5d205b41e1e6efbf641b1587fde5"}
Jul 02 18:39:48.187604 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.187502    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" event={"ID":"a245f3bd-e1b2-4adb-a2d2-f38e34b31858","Type":"ContainerStarted","Data":"ba62d6dd2e4cb3bba0813872a99cf6f9f4bba0bd582743b13c6d55d13e67d6ff"}
Jul 02 18:39:48.188041 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.187709    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:48.189553 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.189526    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" event={"ID":"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97","Type":"ContainerStarted","Data":"bcf8c385226a2d935df8395d7de5d056ea7c1d5a42fc6ea5614948bf215839be"}
Jul 02 18:39:48.189690 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.189584    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:48.189690 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.189636    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:48.191461 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.191414    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 18:39:48.191589 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.191501    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" event={"ID":"5e792528-8683-400b-a079-aa659d1a1c5f","Type":"ContainerStarted","Data":"0925c1d3f4c7116fcd4b843afd7c14189a73c316df832731e13d9f159fd20ea3"}
Jul 02 18:39:48.191804 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.191787    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:39:48.191914 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.191834    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:39:48.193212 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.193166    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 18:39:48.193763 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.193743    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" event={"ID":"1b12d2ba-3425-4dc3-b91d-e709f8436b48","Type":"ContainerStarted","Data":"945841afc998c6c38d83f2bab074fa014dde80846dc804cb1763436c5ffa884c"}
Jul 02 18:39:48.193873 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.193859    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:48.193945 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.193909    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:48.195007 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.194986    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:39:48.208155 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.208107    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" podStartSLOduration=3.449691668 podStartE2EDuration="37.208091461s" totalImagesPullingTime="29.870233543s" totalInitContainerRuntime="3.88816625s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:39:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:39:48.205516572 +0000 UTC m=+580.275144119" watchObservedRunningTime="2026-07-02 18:39:48.208091461 +0000 UTC m=+580.277718989"
Jul 02 18:39:48.224493 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.224447    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" podStartSLOduration=12.375713099 podStartE2EDuration="37.224429653s" totalImagesPullingTime="21.276393167s" totalInitContainerRuntime="3.572323387s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:39:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:39:48.222828021 +0000 UTC m=+580.292455600" watchObservedRunningTime="2026-07-02 18:39:48.224429653 +0000 UTC m=+580.294057181"
Jul 02 18:39:48.241128 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.241079    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" podStartSLOduration=2.48663133 podStartE2EDuration="36.241063828s" totalImagesPullingTime="29.809122084s" totalInitContainerRuntime="3.945310414s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:39:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:39:48.240105111 +0000 UTC m=+580.309732647" watchObservedRunningTime="2026-07-02 18:39:48.241063828 +0000 UTC m=+580.310691355"
Jul 02 18:39:48.257775 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:48.257728    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" podStartSLOduration=1.9547753399999999 podStartE2EDuration="37.257710804s" totalImagesPullingTime="35.302935464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:39:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:39:48.256300909 +0000 UTC m=+580.325928437" watchObservedRunningTime="2026-07-02 18:39:48.257710804 +0000 UTC m=+580.327338333"
Jul 02 18:39:49.198101 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:49.198054    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 18:39:49.198609 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:49.198054    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:39:49.198609 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:49.198160    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 18:39:49.198609 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:49.198561    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:49.199889 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:49.199865    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 18:39:50.202245 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:50.202187    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 18:39:54.203747 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:54.203715    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:39:54.204424 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:54.204387    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 18:39:54.204584 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:54.204456    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:39:54.204584 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:54.204547    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:39:54.205005 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:54.204966    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:39:54.205131 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:54.205060    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 18:39:55.206445 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:55.206414    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:39:55.207012 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:39:55.206970    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 18:40:04.205286 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:04.205246    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:40:04.205743 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:04.205254    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 18:40:04.205743 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:04.205254    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 18:40:05.207004 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:05.206966    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 18:40:08.361499 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:08.361470    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 18:40:08.363307 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:08.363285    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 18:40:08.369415 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:08.369395    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 18:40:08.371442 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:08.371424    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 18:40:14.205017 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:14.204973    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:40:14.325841 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:14.204973    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 18:40:14.325841 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:14.204974    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 18:40:15.206948 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:15.206909    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 18:40:24.204704 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:24.204663    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 18:40:24.205159 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:24.204963    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:40:24.205159 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:24.205010    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 18:40:25.207083 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:25.207037    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 18:40:25.740781 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:25.740742    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:40:34.204962 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:34.204864    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:40:34.205431 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:34.204995    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 18:40:34.205431 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:34.205309    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:40:35.207616 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:35.207572    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 18:40:41.594508 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:41.594471    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:40:41.694608 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:41.694572    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:40:44.204931 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:44.204886    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 18:40:44.205337 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:44.204897    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:40:45.207187 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:45.207148    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 18:40:51.758031 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:51.757999    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:40:51.883273 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:51.883239    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:40:51.967149 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:51.967110    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:40:51.984636 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:51.984608    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"]
Jul 02 18:40:51.988292 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:51.988273    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:40:51.991390 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:51.991372    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a5373-kube-rbac-proxy-sar-config\""
Jul 02 18:40:51.991515 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:51.991372    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a5373-predictor-serving-cert\""
Jul 02 18:40:51.998488 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:51.998466    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"]
Jul 02 18:40:52.039167 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.039081    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"]
Jul 02 18:40:52.039432 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.039406    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kserve-container" containerID="cri-o://dfa22108b05967bf93c60f03fa39e3dbc2394a3e7419a3dd03b7f80c5c76fe03" gracePeriod=30
Jul 02 18:40:52.039559 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.039532    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kube-rbac-proxy" containerID="cri-o://0925c1d3f4c7116fcd4b843afd7c14189a73c316df832731e13d9f159fd20ea3" gracePeriod=30
Jul 02 18:40:52.043102 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.043029    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:40:52.079512 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.079478    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbtxd\" (UniqueName: \"kubernetes.io/projected/d2698acf-ea23-4665-925e-de1859ba6370-kube-api-access-xbtxd\") pod \"error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn\" (UID: \"d2698acf-ea23-4665-925e-de1859ba6370\") " pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:40:52.079617 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.079557    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2698acf-ea23-4665-925e-de1859ba6370-proxy-tls\") pod \"error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn\" (UID: \"d2698acf-ea23-4665-925e-de1859ba6370\") " pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:40:52.079617 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.079579    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2698acf-ea23-4665-925e-de1859ba6370-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn\" (UID: \"d2698acf-ea23-4665-925e-de1859ba6370\") " pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:40:52.142905 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.142875    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:40:52.180826 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.180796    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2698acf-ea23-4665-925e-de1859ba6370-proxy-tls\") pod \"error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn\" (UID: \"d2698acf-ea23-4665-925e-de1859ba6370\") " pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:40:52.180962 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.180835    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2698acf-ea23-4665-925e-de1859ba6370-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn\" (UID: \"d2698acf-ea23-4665-925e-de1859ba6370\") " pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:40:52.180962 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.180911    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbtxd\" (UniqueName: \"kubernetes.io/projected/d2698acf-ea23-4665-925e-de1859ba6370-kube-api-access-xbtxd\") pod \"error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn\" (UID: \"d2698acf-ea23-4665-925e-de1859ba6370\") " pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:40:52.181072 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:40:52.180968    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-a5373-predictor-serving-cert: secret "error-404-isvc-a5373-predictor-serving-cert" not found
Jul 02 18:40:52.181072 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:40:52.181041    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2698acf-ea23-4665-925e-de1859ba6370-proxy-tls podName:d2698acf-ea23-4665-925e-de1859ba6370 nodeName:}" failed. No retries permitted until 2026-07-02 18:40:52.681020152 +0000 UTC m=+644.750647664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d2698acf-ea23-4665-925e-de1859ba6370-proxy-tls") pod "error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" (UID: "d2698acf-ea23-4665-925e-de1859ba6370") : secret "error-404-isvc-a5373-predictor-serving-cert" not found
Jul 02 18:40:52.181489 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.181470    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2698acf-ea23-4665-925e-de1859ba6370-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn\" (UID: \"d2698acf-ea23-4665-925e-de1859ba6370\") " pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:40:52.189393 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.189373    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbtxd\" (UniqueName: \"kubernetes.io/projected/d2698acf-ea23-4665-925e-de1859ba6370-kube-api-access-xbtxd\") pod \"error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn\" (UID: \"d2698acf-ea23-4665-925e-de1859ba6370\") " pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:40:52.271746 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.271712    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:40:52.480010 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.479977    2575 generic.go:356] "Generic (PLEG): container finished" podID="5e792528-8683-400b-a079-aa659d1a1c5f" containerID="0925c1d3f4c7116fcd4b843afd7c14189a73c316df832731e13d9f159fd20ea3" exitCode=2
Jul 02 18:40:52.480176 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.480044    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" event={"ID":"5e792528-8683-400b-a079-aa659d1a1c5f","Type":"ContainerDied","Data":"0925c1d3f4c7116fcd4b843afd7c14189a73c316df832731e13d9f159fd20ea3"}
Jul 02 18:40:52.684750 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.684717    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2698acf-ea23-4665-925e-de1859ba6370-proxy-tls\") pod \"error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn\" (UID: \"d2698acf-ea23-4665-925e-de1859ba6370\") " pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:40:52.687053 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.687023    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2698acf-ea23-4665-925e-de1859ba6370-proxy-tls\") pod \"error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn\" (UID: \"d2698acf-ea23-4665-925e-de1859ba6370\") " pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:40:52.899122 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:52.899085    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:40:53.031863 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:53.031826    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"]
Jul 02 18:40:53.035223 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:40:53.035166    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2698acf_ea23_4665_925e_de1859ba6370.slice/crio-76cb855194a94629a6070cfcb0ee4b793251874839746f9aae691d0d60541282 WatchSource:0}: Error finding container 76cb855194a94629a6070cfcb0ee4b793251874839746f9aae691d0d60541282: Status 404 returned error can't find the container with id 76cb855194a94629a6070cfcb0ee4b793251874839746f9aae691d0d60541282
Jul 02 18:40:53.485776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:53.485699    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" event={"ID":"d2698acf-ea23-4665-925e-de1859ba6370","Type":"ContainerStarted","Data":"3c2bff469783c06f37fe0229701463574d5f4076ed7eadb30b3b913c0abbd699"}
Jul 02 18:40:53.485776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:53.485753    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" event={"ID":"d2698acf-ea23-4665-925e-de1859ba6370","Type":"ContainerStarted","Data":"089bae17bf8a8dac0376ae3a0e1ec586bf5adeb0a702cdd873ae290023fca012"}
Jul 02 18:40:53.485776 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:53.485775    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" event={"ID":"d2698acf-ea23-4665-925e-de1859ba6370","Type":"ContainerStarted","Data":"76cb855194a94629a6070cfcb0ee4b793251874839746f9aae691d0d60541282"}
Jul 02 18:40:53.485991 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:53.485807    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:40:53.505593 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:53.505537    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" podStartSLOduration=2.505520341 podStartE2EDuration="2.505520341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:40:53.503756934 +0000 UTC m=+645.573384483" watchObservedRunningTime="2026-07-02 18:40:53.505520341 +0000 UTC m=+645.575147869"
Jul 02 18:40:54.198525 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:54.198466    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 02 18:40:54.205174 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:54.205139    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 18:40:54.205174 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:54.205160    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 18:40:54.205364 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:54.205160    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:40:54.490070 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:54.489990    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:40:54.491052 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:54.491025    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" podUID="d2698acf-ea23-4665-925e-de1859ba6370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 02 18:40:55.208073 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.208041    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:40:55.484773 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.484752    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:40:55.494807 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.494766    2575 generic.go:356] "Generic (PLEG): container finished" podID="5e792528-8683-400b-a079-aa659d1a1c5f" containerID="dfa22108b05967bf93c60f03fa39e3dbc2394a3e7419a3dd03b7f80c5c76fe03" exitCode=0
Jul 02 18:40:55.494919 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.494817    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"
Jul 02 18:40:55.494919 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.494842    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" event={"ID":"5e792528-8683-400b-a079-aa659d1a1c5f","Type":"ContainerDied","Data":"dfa22108b05967bf93c60f03fa39e3dbc2394a3e7419a3dd03b7f80c5c76fe03"}
Jul 02 18:40:55.494919 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.494903    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6" event={"ID":"5e792528-8683-400b-a079-aa659d1a1c5f","Type":"ContainerDied","Data":"6f29967eb7df0227a882b4f6d9e0ece3898bafeac6672353ba7d0d591e658f7c"}
Jul 02 18:40:55.495085 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.494932    2575 scope.go:117] "RemoveContainer" containerID="0925c1d3f4c7116fcd4b843afd7c14189a73c316df832731e13d9f159fd20ea3"
Jul 02 18:40:55.495385 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.495363    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" podUID="d2698acf-ea23-4665-925e-de1859ba6370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 02 18:40:55.503250 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.503227    2575 scope.go:117] "RemoveContainer" containerID="dfa22108b05967bf93c60f03fa39e3dbc2394a3e7419a3dd03b7f80c5c76fe03"
Jul 02 18:40:55.508860 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.508838    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dh68f\" (UniqueName: \"kubernetes.io/projected/5e792528-8683-400b-a079-aa659d1a1c5f-kube-api-access-dh68f\") pod \"5e792528-8683-400b-a079-aa659d1a1c5f\" (UID: \"5e792528-8683-400b-a079-aa659d1a1c5f\") "
Jul 02 18:40:55.508965 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.508891    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e792528-8683-400b-a079-aa659d1a1c5f-proxy-tls\") pod \"5e792528-8683-400b-a079-aa659d1a1c5f\" (UID: \"5e792528-8683-400b-a079-aa659d1a1c5f\") "
Jul 02 18:40:55.508965 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.508910    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e792528-8683-400b-a079-aa659d1a1c5f-kube-rbac-proxy-sar-config\") pod \"5e792528-8683-400b-a079-aa659d1a1c5f\" (UID: \"5e792528-8683-400b-a079-aa659d1a1c5f\") "
Jul 02 18:40:55.509362 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.509332    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e792528-8683-400b-a079-aa659d1a1c5f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5e792528-8683-400b-a079-aa659d1a1c5f" (UID: "5e792528-8683-400b-a079-aa659d1a1c5f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:40:55.511227 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.511176    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e792528-8683-400b-a079-aa659d1a1c5f-kube-api-access-dh68f" (OuterVolumeSpecName: "kube-api-access-dh68f") pod "5e792528-8683-400b-a079-aa659d1a1c5f" (UID: "5e792528-8683-400b-a079-aa659d1a1c5f"). InnerVolumeSpecName "kube-api-access-dh68f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:40:55.511316 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.511218    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e792528-8683-400b-a079-aa659d1a1c5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e792528-8683-400b-a079-aa659d1a1c5f" (UID: "5e792528-8683-400b-a079-aa659d1a1c5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:40:55.512034 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.512018    2575 scope.go:117] "RemoveContainer" containerID="0925c1d3f4c7116fcd4b843afd7c14189a73c316df832731e13d9f159fd20ea3"
Jul 02 18:40:55.512457 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:40:55.512432    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0925c1d3f4c7116fcd4b843afd7c14189a73c316df832731e13d9f159fd20ea3\": container with ID starting with 0925c1d3f4c7116fcd4b843afd7c14189a73c316df832731e13d9f159fd20ea3 not found: ID does not exist" containerID="0925c1d3f4c7116fcd4b843afd7c14189a73c316df832731e13d9f159fd20ea3"
Jul 02 18:40:55.512530 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.512469    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0925c1d3f4c7116fcd4b843afd7c14189a73c316df832731e13d9f159fd20ea3"} err="failed to get container status \"0925c1d3f4c7116fcd4b843afd7c14189a73c316df832731e13d9f159fd20ea3\": rpc error: code = NotFound desc = could not find container \"0925c1d3f4c7116fcd4b843afd7c14189a73c316df832731e13d9f159fd20ea3\": container with ID starting with 0925c1d3f4c7116fcd4b843afd7c14189a73c316df832731e13d9f159fd20ea3 not found: ID does not exist"
Jul 02 18:40:55.512530 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.512495    2575 scope.go:117] "RemoveContainer" containerID="dfa22108b05967bf93c60f03fa39e3dbc2394a3e7419a3dd03b7f80c5c76fe03"
Jul 02 18:40:55.512759 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:40:55.512738    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa22108b05967bf93c60f03fa39e3dbc2394a3e7419a3dd03b7f80c5c76fe03\": container with ID starting with dfa22108b05967bf93c60f03fa39e3dbc2394a3e7419a3dd03b7f80c5c76fe03 not found: ID does not exist" containerID="dfa22108b05967bf93c60f03fa39e3dbc2394a3e7419a3dd03b7f80c5c76fe03"
Jul 02 18:40:55.512817 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.512771    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa22108b05967bf93c60f03fa39e3dbc2394a3e7419a3dd03b7f80c5c76fe03"} err="failed to get container status \"dfa22108b05967bf93c60f03fa39e3dbc2394a3e7419a3dd03b7f80c5c76fe03\": rpc error: code = NotFound desc = could not find container \"dfa22108b05967bf93c60f03fa39e3dbc2394a3e7419a3dd03b7f80c5c76fe03\": container with ID starting with dfa22108b05967bf93c60f03fa39e3dbc2394a3e7419a3dd03b7f80c5c76fe03 not found: ID does not exist"
Jul 02 18:40:55.610461 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.610424    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh68f\" (UniqueName: \"kubernetes.io/projected/5e792528-8683-400b-a079-aa659d1a1c5f-kube-api-access-dh68f\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:40:55.610461 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.610456    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e792528-8683-400b-a079-aa659d1a1c5f-proxy-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:40:55.610461 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.610470    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e792528-8683-400b-a079-aa659d1a1c5f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:40:55.817907 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.817871    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"]
Jul 02 18:40:55.821429 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:55.821399    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb242-predictor-7d85f5f688-cxpx6"]
Jul 02 18:40:56.489147 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:56.489116    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" path="/var/lib/kubelet/pods/5e792528-8683-400b-a079-aa659d1a1c5f/volumes"
Jul 02 18:40:59.846365 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:40:59.846312    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:41:00.500163 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:00.500135    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:41:00.500741 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:00.500716    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" podUID="d2698acf-ea23-4665-925e-de1859ba6370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 02 18:41:01.512909 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:01.512876    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:41:04.206218 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:04.206164    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:41:04.206615 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:04.206358    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:41:10.501567 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:10.501521    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" podUID="d2698acf-ea23-4665-925e-de1859ba6370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 02 18:41:20.501379 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:20.501339    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" podUID="d2698acf-ea23-4665-925e-de1859ba6370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 02 18:41:30.501379 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:30.501332    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" podUID="d2698acf-ea23-4665-925e-de1859ba6370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 02 18:41:31.563499 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:31.563463    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:41:32.133433 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:32.133398    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:41:39.805230 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:39.805144    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:41:40.502096 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:40.502066    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:41:41.743642 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.743611    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"]
Jul 02 18:41:41.744097 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.743913    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kserve-container" containerID="cri-o://0e8e45a366d8886a9489c7b62dc377d3b595fb19af7594cc27b05d9a6b527a1d" gracePeriod=30
Jul 02 18:41:41.744097 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.744014    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kube-rbac-proxy" containerID="cri-o://ba62d6dd2e4cb3bba0813872a99cf6f9f4bba0bd582743b13c6d55d13e67d6ff" gracePeriod=30
Jul 02 18:41:41.745347 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.745324    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:41:41.794711 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.794680    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"]
Jul 02 18:41:41.794930 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.794895    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kserve-container" containerID="cri-o://b8a770043c59ee4127446981377122c06bbd35e1401da14dbc4eef73abb23e48" gracePeriod=30
Jul 02 18:41:41.795069 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.794957    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kube-rbac-proxy" containerID="cri-o://945841afc998c6c38d83f2bab074fa014dde80846dc804cb1763436c5ffa884c" gracePeriod=30
Jul 02 18:41:41.820056 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.820009    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:41:41.846763 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.846730    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"]
Jul 02 18:41:41.847021 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.846991    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kserve-container" containerID="cri-o://69bb3c286363b8b7222474e2f69ff7ddad4d5d205b41e1e6efbf641b1587fde5" gracePeriod=30
Jul 02 18:41:41.847143 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.847035    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kube-rbac-proxy" containerID="cri-o://bcf8c385226a2d935df8395d7de5d056ea7c1d5a42fc6ea5614948bf215839be" gracePeriod=30
Jul 02 18:41:41.910241 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.910211    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:41:41.923621 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.923592    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"]
Jul 02 18:41:41.923935 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.923923    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kube-rbac-proxy"
Jul 02 18:41:41.923979 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.923936    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kube-rbac-proxy"
Jul 02 18:41:41.923979 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.923958    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kserve-container"
Jul 02 18:41:41.923979 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.923964    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kserve-container"
Jul 02 18:41:41.924070 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.924018    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kserve-container"
Jul 02 18:41:41.924070 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.924030    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e792528-8683-400b-a079-aa659d1a1c5f" containerName="kube-rbac-proxy"
Jul 02 18:41:41.927014 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.926992    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:41.929786 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.929759    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e8b76-predictor-serving-cert\""
Jul 02 18:41:41.929989 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.929970    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e8b76-kube-rbac-proxy-sar-config\""
Jul 02 18:41:41.936352 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:41.936329    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"]
Jul 02 18:41:42.011646 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.011516    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:41:42.030564 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.030529    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fefe989b-4300-4eb7-bb66-816c42550acc-proxy-tls\") pod \"error-404-isvc-e8b76-predictor-756c9998f6-2lnkl\" (UID: \"fefe989b-4300-4eb7-bb66-816c42550acc\") " pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:42.030774 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.030577    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fefe989b-4300-4eb7-bb66-816c42550acc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e8b76-predictor-756c9998f6-2lnkl\" (UID: \"fefe989b-4300-4eb7-bb66-816c42550acc\") " pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:42.030774 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.030696    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvqkp\" (UniqueName: \"kubernetes.io/projected/fefe989b-4300-4eb7-bb66-816c42550acc-kube-api-access-lvqkp\") pod \"error-404-isvc-e8b76-predictor-756c9998f6-2lnkl\" (UID: \"fefe989b-4300-4eb7-bb66-816c42550acc\") " pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:42.110601 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.110567    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:41:42.131759 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.131713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fefe989b-4300-4eb7-bb66-816c42550acc-proxy-tls\") pod \"error-404-isvc-e8b76-predictor-756c9998f6-2lnkl\" (UID: \"fefe989b-4300-4eb7-bb66-816c42550acc\") " pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:42.131759 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.131767    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fefe989b-4300-4eb7-bb66-816c42550acc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e8b76-predictor-756c9998f6-2lnkl\" (UID: \"fefe989b-4300-4eb7-bb66-816c42550acc\") " pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:42.132024 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.131806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvqkp\" (UniqueName: \"kubernetes.io/projected/fefe989b-4300-4eb7-bb66-816c42550acc-kube-api-access-lvqkp\") pod \"error-404-isvc-e8b76-predictor-756c9998f6-2lnkl\" (UID: \"fefe989b-4300-4eb7-bb66-816c42550acc\") " pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:42.132024 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:41:42.131881    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-serving-cert: secret "error-404-isvc-e8b76-predictor-serving-cert" not found
Jul 02 18:41:42.132024 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:41:42.131958    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fefe989b-4300-4eb7-bb66-816c42550acc-proxy-tls podName:fefe989b-4300-4eb7-bb66-816c42550acc nodeName:}" failed. No retries permitted until 2026-07-02 18:41:42.631935889 +0000 UTC m=+694.701563415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fefe989b-4300-4eb7-bb66-816c42550acc-proxy-tls") pod "error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" (UID: "fefe989b-4300-4eb7-bb66-816c42550acc") : secret "error-404-isvc-e8b76-predictor-serving-cert" not found
Jul 02 18:41:42.132666 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.132635    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fefe989b-4300-4eb7-bb66-816c42550acc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e8b76-predictor-756c9998f6-2lnkl\" (UID: \"fefe989b-4300-4eb7-bb66-816c42550acc\") " pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:42.140809 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.140788    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvqkp\" (UniqueName: \"kubernetes.io/projected/fefe989b-4300-4eb7-bb66-816c42550acc-kube-api-access-lvqkp\") pod \"error-404-isvc-e8b76-predictor-756c9998f6-2lnkl\" (UID: \"fefe989b-4300-4eb7-bb66-816c42550acc\") " pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:42.387147 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.387117    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:41:42.638762 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.638658    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fefe989b-4300-4eb7-bb66-816c42550acc-proxy-tls\") pod \"error-404-isvc-e8b76-predictor-756c9998f6-2lnkl\" (UID: \"fefe989b-4300-4eb7-bb66-816c42550acc\") " pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:42.641115 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.641081    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fefe989b-4300-4eb7-bb66-816c42550acc-proxy-tls\") pod \"error-404-isvc-e8b76-predictor-756c9998f6-2lnkl\" (UID: \"fefe989b-4300-4eb7-bb66-816c42550acc\") " pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:42.714333 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.714297    2575 generic.go:356] "Generic (PLEG): container finished" podID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerID="ba62d6dd2e4cb3bba0813872a99cf6f9f4bba0bd582743b13c6d55d13e67d6ff" exitCode=2
Jul 02 18:41:42.714520 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.714372    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" event={"ID":"a245f3bd-e1b2-4adb-a2d2-f38e34b31858","Type":"ContainerDied","Data":"ba62d6dd2e4cb3bba0813872a99cf6f9f4bba0bd582743b13c6d55d13e67d6ff"}
Jul 02 18:41:42.716277 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.716255    2575 generic.go:356] "Generic (PLEG): container finished" podID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerID="bcf8c385226a2d935df8395d7de5d056ea7c1d5a42fc6ea5614948bf215839be" exitCode=2
Jul 02 18:41:42.716379 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.716324    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" event={"ID":"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97","Type":"ContainerDied","Data":"bcf8c385226a2d935df8395d7de5d056ea7c1d5a42fc6ea5614948bf215839be"}
Jul 02 18:41:42.718304 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.718283    2575 generic.go:356] "Generic (PLEG): container finished" podID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerID="945841afc998c6c38d83f2bab074fa014dde80846dc804cb1763436c5ffa884c" exitCode=2
Jul 02 18:41:42.718396 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.718330    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" event={"ID":"1b12d2ba-3425-4dc3-b91d-e709f8436b48","Type":"ContainerDied","Data":"945841afc998c6c38d83f2bab074fa014dde80846dc804cb1763436c5ffa884c"}
Jul 02 18:41:42.837784 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.837739    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:42.966406 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:42.966380    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"]
Jul 02 18:41:42.968813 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:41:42.968782    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefe989b_4300_4eb7_bb66_816c42550acc.slice/crio-4f15c033f23ac84f9b202035d5e7a48bda9877e1555e962325258d6a61fcd5e7 WatchSource:0}: Error finding container 4f15c033f23ac84f9b202035d5e7a48bda9877e1555e962325258d6a61fcd5e7: Status 404 returned error can't find the container with id 4f15c033f23ac84f9b202035d5e7a48bda9877e1555e962325258d6a61fcd5e7
Jul 02 18:41:43.725357 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:43.725316    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" event={"ID":"fefe989b-4300-4eb7-bb66-816c42550acc","Type":"ContainerStarted","Data":"71c387d4a80f68e49d1290b4421d3494a5c587849cef51784c9b4ee1ddc563b5"}
Jul 02 18:41:43.725569 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:43.725377    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" event={"ID":"fefe989b-4300-4eb7-bb66-816c42550acc","Type":"ContainerStarted","Data":"d8e6b2fcbf561946d8c0e22874aa91c4884238f8ed0cdaa72ffdc7eb8e324351"}
Jul 02 18:41:43.725569 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:43.725406    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" event={"ID":"fefe989b-4300-4eb7-bb66-816c42550acc","Type":"ContainerStarted","Data":"4f15c033f23ac84f9b202035d5e7a48bda9877e1555e962325258d6a61fcd5e7"}
Jul 02 18:41:43.725569 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:43.725489    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:43.743373 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:43.743328    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" podStartSLOduration=2.743312583 podStartE2EDuration="2.743312583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:41:43.742559972 +0000 UTC m=+695.812187517" watchObservedRunningTime="2026-07-02 18:41:43.743312583 +0000 UTC m=+695.812940112"
Jul 02 18:41:44.198289 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:44.198248    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 02 18:41:44.198697 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:44.198250    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 02 18:41:44.205715 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:44.205683    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:41:44.205825 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:44.205689    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 18:41:44.731030 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:44.730992    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:44.732618 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:44.732579    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 02 18:41:45.202521 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.202483    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 02 18:41:45.207886 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.207858    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 18:41:45.701538 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.701513    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:41:45.737834 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.737796    2575 generic.go:356] "Generic (PLEG): container finished" podID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerID="0e8e45a366d8886a9489c7b62dc377d3b595fb19af7594cc27b05d9a6b527a1d" exitCode=0
Jul 02 18:41:45.738007 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.737855    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"
Jul 02 18:41:45.738007 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.737869    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" event={"ID":"a245f3bd-e1b2-4adb-a2d2-f38e34b31858","Type":"ContainerDied","Data":"0e8e45a366d8886a9489c7b62dc377d3b595fb19af7594cc27b05d9a6b527a1d"}
Jul 02 18:41:45.738007 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.737940    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq" event={"ID":"a245f3bd-e1b2-4adb-a2d2-f38e34b31858","Type":"ContainerDied","Data":"e8775149fef048802f0d4bd4a9d91e4b4ba19f17a836d6cc8eef0f2b77036659"}
Jul 02 18:41:45.738007 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.737974    2575 scope.go:117] "RemoveContainer" containerID="ba62d6dd2e4cb3bba0813872a99cf6f9f4bba0bd582743b13c6d55d13e67d6ff"
Jul 02 18:41:45.738499 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.738468    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 02 18:41:45.746027 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.745999    2575 scope.go:117] "RemoveContainer" containerID="0e8e45a366d8886a9489c7b62dc377d3b595fb19af7594cc27b05d9a6b527a1d"
Jul 02 18:41:45.753136 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.753116    2575 scope.go:117] "RemoveContainer" containerID="b1f34d84a5d2d9a91241e3b45ed66a0a25f645b905e9bff1f862b0a469896c9b"
Jul 02 18:41:45.760292 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.760278    2575 scope.go:117] "RemoveContainer" containerID="ba62d6dd2e4cb3bba0813872a99cf6f9f4bba0bd582743b13c6d55d13e67d6ff"
Jul 02 18:41:45.760528 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:41:45.760509    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba62d6dd2e4cb3bba0813872a99cf6f9f4bba0bd582743b13c6d55d13e67d6ff\": container with ID starting with ba62d6dd2e4cb3bba0813872a99cf6f9f4bba0bd582743b13c6d55d13e67d6ff not found: ID does not exist" containerID="ba62d6dd2e4cb3bba0813872a99cf6f9f4bba0bd582743b13c6d55d13e67d6ff"
Jul 02 18:41:45.760571 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.760539    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba62d6dd2e4cb3bba0813872a99cf6f9f4bba0bd582743b13c6d55d13e67d6ff"} err="failed to get container status \"ba62d6dd2e4cb3bba0813872a99cf6f9f4bba0bd582743b13c6d55d13e67d6ff\": rpc error: code = NotFound desc = could not find container \"ba62d6dd2e4cb3bba0813872a99cf6f9f4bba0bd582743b13c6d55d13e67d6ff\": container with ID starting with ba62d6dd2e4cb3bba0813872a99cf6f9f4bba0bd582743b13c6d55d13e67d6ff not found: ID does not exist"
Jul 02 18:41:45.760571 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.760560    2575 scope.go:117] "RemoveContainer" containerID="0e8e45a366d8886a9489c7b62dc377d3b595fb19af7594cc27b05d9a6b527a1d"
Jul 02 18:41:45.760788 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:41:45.760772    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e8e45a366d8886a9489c7b62dc377d3b595fb19af7594cc27b05d9a6b527a1d\": container with ID starting with 0e8e45a366d8886a9489c7b62dc377d3b595fb19af7594cc27b05d9a6b527a1d not found: ID does not exist" containerID="0e8e45a366d8886a9489c7b62dc377d3b595fb19af7594cc27b05d9a6b527a1d"
Jul 02 18:41:45.760829 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.760794    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e8e45a366d8886a9489c7b62dc377d3b595fb19af7594cc27b05d9a6b527a1d"} err="failed to get container status \"0e8e45a366d8886a9489c7b62dc377d3b595fb19af7594cc27b05d9a6b527a1d\": rpc error: code = NotFound desc = could not find container \"0e8e45a366d8886a9489c7b62dc377d3b595fb19af7594cc27b05d9a6b527a1d\": container with ID starting with 0e8e45a366d8886a9489c7b62dc377d3b595fb19af7594cc27b05d9a6b527a1d not found: ID does not exist"
Jul 02 18:41:45.760829 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.760809    2575 scope.go:117] "RemoveContainer" containerID="b1f34d84a5d2d9a91241e3b45ed66a0a25f645b905e9bff1f862b0a469896c9b"
Jul 02 18:41:45.761012 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:41:45.760996    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f34d84a5d2d9a91241e3b45ed66a0a25f645b905e9bff1f862b0a469896c9b\": container with ID starting with b1f34d84a5d2d9a91241e3b45ed66a0a25f645b905e9bff1f862b0a469896c9b not found: ID does not exist" containerID="b1f34d84a5d2d9a91241e3b45ed66a0a25f645b905e9bff1f862b0a469896c9b"
Jul 02 18:41:45.761054 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.761017    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f34d84a5d2d9a91241e3b45ed66a0a25f645b905e9bff1f862b0a469896c9b"} err="failed to get container status \"b1f34d84a5d2d9a91241e3b45ed66a0a25f645b905e9bff1f862b0a469896c9b\": rpc error: code = NotFound desc = could not find container \"b1f34d84a5d2d9a91241e3b45ed66a0a25f645b905e9bff1f862b0a469896c9b\": container with ID starting with b1f34d84a5d2d9a91241e3b45ed66a0a25f645b905e9bff1f862b0a469896c9b not found: ID does not exist"
Jul 02 18:41:45.767273 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.767254    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plcsr\" (UniqueName: \"kubernetes.io/projected/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kube-api-access-plcsr\") pod \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") "
Jul 02 18:41:45.767343 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.767307    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kube-rbac-proxy-sar-config\") pod \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") "
Jul 02 18:41:45.767343 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.767338    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-proxy-tls\") pod \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") "
Jul 02 18:41:45.767436 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.767372    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kserve-provision-location\") pod \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") "
Jul 02 18:41:45.767436 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.767389    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-cabundle-cert\") pod \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\" (UID: \"a245f3bd-e1b2-4adb-a2d2-f38e34b31858\") "
Jul 02 18:41:45.767681 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.767657    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a245f3bd-e1b2-4adb-a2d2-f38e34b31858" (UID: "a245f3bd-e1b2-4adb-a2d2-f38e34b31858"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:41:45.767753 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.767709    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a245f3bd-e1b2-4adb-a2d2-f38e34b31858" (UID: "a245f3bd-e1b2-4adb-a2d2-f38e34b31858"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:41:45.767794 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.767740    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a245f3bd-e1b2-4adb-a2d2-f38e34b31858" (UID: "a245f3bd-e1b2-4adb-a2d2-f38e34b31858"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:41:45.769166 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.769137    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kube-api-access-plcsr" (OuterVolumeSpecName: "kube-api-access-plcsr") pod "a245f3bd-e1b2-4adb-a2d2-f38e34b31858" (UID: "a245f3bd-e1b2-4adb-a2d2-f38e34b31858"). InnerVolumeSpecName "kube-api-access-plcsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:41:45.769299 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.769284    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a245f3bd-e1b2-4adb-a2d2-f38e34b31858" (UID: "a245f3bd-e1b2-4adb-a2d2-f38e34b31858"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:41:45.868337 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.868308    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kserve-provision-location\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:45.868337 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.868335    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-cabundle-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:45.868535 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.868348    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plcsr\" (UniqueName: \"kubernetes.io/projected/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kube-api-access-plcsr\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:45.868535 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.868360    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:45.868535 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:45.868374    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a245f3bd-e1b2-4adb-a2d2-f38e34b31858-proxy-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:46.059706 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:46.059676    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"]
Jul 02 18:41:46.065435 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:46.065413    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-c5zrq"]
Jul 02 18:41:46.488238 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:46.488182    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" path="/var/lib/kubelet/pods/a245f3bd-e1b2-4adb-a2d2-f38e34b31858/volumes"
Jul 02 18:41:47.106596 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.106570    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:41:47.179749 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.179721    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ckc7\" (UniqueName: \"kubernetes.io/projected/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kube-api-access-7ckc7\") pod \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") "
Jul 02 18:41:47.179938 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.179765    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-cabundle-cert\") pod \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") "
Jul 02 18:41:47.179938 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.179855    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-proxy-tls\") pod \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") "
Jul 02 18:41:47.179938 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.179899    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kube-rbac-proxy-sar-config\") pod \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") "
Jul 02 18:41:47.179938 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.179927    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kserve-provision-location\") pod \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\" (UID: \"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97\") "
Jul 02 18:41:47.180308 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.180279    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" (UID: "43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:41:47.180425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.180401    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" (UID: "43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:41:47.180587 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.180566    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" (UID: "43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:41:47.181967 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.181920    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kube-api-access-7ckc7" (OuterVolumeSpecName: "kube-api-access-7ckc7") pod "43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" (UID: "43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97"). InnerVolumeSpecName "kube-api-access-7ckc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:41:47.182223 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.182180    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" (UID: "43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:41:47.281451 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.281420    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kserve-provision-location\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:47.281451 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.281446    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ckc7\" (UniqueName: \"kubernetes.io/projected/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kube-api-access-7ckc7\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:47.281451 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.281457    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-cabundle-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:47.281657 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.281466    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-proxy-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:47.281657 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.281474    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:47.453618 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.453594    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:41:47.585033 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.584984    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b12d2ba-3425-4dc3-b91d-e709f8436b48-proxy-tls\") pod \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") "
Jul 02 18:41:47.585505 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.585063    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kserve-provision-location\") pod \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") "
Jul 02 18:41:47.585505 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.585099    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b12d2ba-3425-4dc3-b91d-e709f8436b48-cabundle-cert\") pod \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") "
Jul 02 18:41:47.585505 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.585124    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4c5c\" (UniqueName: \"kubernetes.io/projected/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kube-api-access-z4c5c\") pod \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") "
Jul 02 18:41:47.585505 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.585166    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kube-rbac-proxy-sar-config\") pod \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\" (UID: \"1b12d2ba-3425-4dc3-b91d-e709f8436b48\") "
Jul 02 18:41:47.585505 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.585463    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b12d2ba-3425-4dc3-b91d-e709f8436b48" (UID: "1b12d2ba-3425-4dc3-b91d-e709f8436b48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:41:47.585505 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.585489    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b12d2ba-3425-4dc3-b91d-e709f8436b48-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1b12d2ba-3425-4dc3-b91d-e709f8436b48" (UID: "1b12d2ba-3425-4dc3-b91d-e709f8436b48"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:41:47.585895 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.585528    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1b12d2ba-3425-4dc3-b91d-e709f8436b48" (UID: "1b12d2ba-3425-4dc3-b91d-e709f8436b48"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:41:47.587183 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.587164    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kube-api-access-z4c5c" (OuterVolumeSpecName: "kube-api-access-z4c5c") pod "1b12d2ba-3425-4dc3-b91d-e709f8436b48" (UID: "1b12d2ba-3425-4dc3-b91d-e709f8436b48"). InnerVolumeSpecName "kube-api-access-z4c5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:41:47.587240 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.587187    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b12d2ba-3425-4dc3-b91d-e709f8436b48-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1b12d2ba-3425-4dc3-b91d-e709f8436b48" (UID: "1b12d2ba-3425-4dc3-b91d-e709f8436b48"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:41:47.685996 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.685957    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b12d2ba-3425-4dc3-b91d-e709f8436b48-proxy-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:47.685996 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.685988    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kserve-provision-location\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:47.685996 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.685998    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b12d2ba-3425-4dc3-b91d-e709f8436b48-cabundle-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:47.686262 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.686009    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4c5c\" (UniqueName: \"kubernetes.io/projected/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kube-api-access-z4c5c\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:47.686262 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.686018    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b12d2ba-3425-4dc3-b91d-e709f8436b48-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:41:47.749602 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.749510    2575 generic.go:356] "Generic (PLEG): container finished" podID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerID="b8a770043c59ee4127446981377122c06bbd35e1401da14dbc4eef73abb23e48" exitCode=0
Jul 02 18:41:47.749602 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.749566    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"
Jul 02 18:41:47.749816 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.749596    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" event={"ID":"1b12d2ba-3425-4dc3-b91d-e709f8436b48","Type":"ContainerDied","Data":"b8a770043c59ee4127446981377122c06bbd35e1401da14dbc4eef73abb23e48"}
Jul 02 18:41:47.749816 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.749658    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd" event={"ID":"1b12d2ba-3425-4dc3-b91d-e709f8436b48","Type":"ContainerDied","Data":"eac51bddeea7eb3f01b5c3278e5fc48bca94c0a8115c1c6f2ce8482692017c7a"}
Jul 02 18:41:47.749816 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.749683    2575 scope.go:117] "RemoveContainer" containerID="945841afc998c6c38d83f2bab074fa014dde80846dc804cb1763436c5ffa884c"
Jul 02 18:41:47.751614 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.751590    2575 generic.go:356] "Generic (PLEG): container finished" podID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerID="69bb3c286363b8b7222474e2f69ff7ddad4d5d205b41e1e6efbf641b1587fde5" exitCode=0
Jul 02 18:41:47.751742 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.751642    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"
Jul 02 18:41:47.751742 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.751666    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" event={"ID":"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97","Type":"ContainerDied","Data":"69bb3c286363b8b7222474e2f69ff7ddad4d5d205b41e1e6efbf641b1587fde5"}
Jul 02 18:41:47.751742 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.751717    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr" event={"ID":"43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97","Type":"ContainerDied","Data":"7ea26104425f2ea9190db5f803a22029487122008364c40c2b650ecd4e239332"}
Jul 02 18:41:47.757948 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.757930    2575 scope.go:117] "RemoveContainer" containerID="b8a770043c59ee4127446981377122c06bbd35e1401da14dbc4eef73abb23e48"
Jul 02 18:41:47.767165 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.767149    2575 scope.go:117] "RemoveContainer" containerID="dec416e76609529b7ad6f122c2ce8d9147cf468ef86bcd5c51847d685e8749f9"
Jul 02 18:41:47.773888 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.773868    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"]
Jul 02 18:41:47.774678 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.774660    2575 scope.go:117] "RemoveContainer" containerID="945841afc998c6c38d83f2bab074fa014dde80846dc804cb1763436c5ffa884c"
Jul 02 18:41:47.774967 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:41:47.774944    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945841afc998c6c38d83f2bab074fa014dde80846dc804cb1763436c5ffa884c\": container with ID starting with 945841afc998c6c38d83f2bab074fa014dde80846dc804cb1763436c5ffa884c not found: ID does not exist" containerID="945841afc998c6c38d83f2bab074fa014dde80846dc804cb1763436c5ffa884c"
Jul 02 18:41:47.775059 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.774975    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945841afc998c6c38d83f2bab074fa014dde80846dc804cb1763436c5ffa884c"} err="failed to get container status \"945841afc998c6c38d83f2bab074fa014dde80846dc804cb1763436c5ffa884c\": rpc error: code = NotFound desc = could not find container \"945841afc998c6c38d83f2bab074fa014dde80846dc804cb1763436c5ffa884c\": container with ID starting with 945841afc998c6c38d83f2bab074fa014dde80846dc804cb1763436c5ffa884c not found: ID does not exist"
Jul 02 18:41:47.775059 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.774991    2575 scope.go:117] "RemoveContainer" containerID="b8a770043c59ee4127446981377122c06bbd35e1401da14dbc4eef73abb23e48"
Jul 02 18:41:47.775354 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:41:47.775331    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a770043c59ee4127446981377122c06bbd35e1401da14dbc4eef73abb23e48\": container with ID starting with b8a770043c59ee4127446981377122c06bbd35e1401da14dbc4eef73abb23e48 not found: ID does not exist" containerID="b8a770043c59ee4127446981377122c06bbd35e1401da14dbc4eef73abb23e48"
Jul 02 18:41:47.775491 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.775363    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a770043c59ee4127446981377122c06bbd35e1401da14dbc4eef73abb23e48"} err="failed to get container status \"b8a770043c59ee4127446981377122c06bbd35e1401da14dbc4eef73abb23e48\": rpc error: code = NotFound desc = could not find container \"b8a770043c59ee4127446981377122c06bbd35e1401da14dbc4eef73abb23e48\": container with ID starting with b8a770043c59ee4127446981377122c06bbd35e1401da14dbc4eef73abb23e48 not found: ID does not exist"
Jul 02 18:41:47.775491 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.775382    2575 scope.go:117] "RemoveContainer" containerID="dec416e76609529b7ad6f122c2ce8d9147cf468ef86bcd5c51847d685e8749f9"
Jul 02 18:41:47.776278 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:41:47.775902    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec416e76609529b7ad6f122c2ce8d9147cf468ef86bcd5c51847d685e8749f9\": container with ID starting with dec416e76609529b7ad6f122c2ce8d9147cf468ef86bcd5c51847d685e8749f9 not found: ID does not exist" containerID="dec416e76609529b7ad6f122c2ce8d9147cf468ef86bcd5c51847d685e8749f9"
Jul 02 18:41:47.776278 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.775930    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec416e76609529b7ad6f122c2ce8d9147cf468ef86bcd5c51847d685e8749f9"} err="failed to get container status \"dec416e76609529b7ad6f122c2ce8d9147cf468ef86bcd5c51847d685e8749f9\": rpc error: code = NotFound desc = could not find container \"dec416e76609529b7ad6f122c2ce8d9147cf468ef86bcd5c51847d685e8749f9\": container with ID starting with dec416e76609529b7ad6f122c2ce8d9147cf468ef86bcd5c51847d685e8749f9 not found: ID does not exist"
Jul 02 18:41:47.776278 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.775948    2575 scope.go:117] "RemoveContainer" containerID="bcf8c385226a2d935df8395d7de5d056ea7c1d5a42fc6ea5614948bf215839be"
Jul 02 18:41:47.778036 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.778021    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-mlggd"]
Jul 02 18:41:47.783135 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.783119    2575 scope.go:117] "RemoveContainer" containerID="69bb3c286363b8b7222474e2f69ff7ddad4d5d205b41e1e6efbf641b1587fde5"
Jul 02 18:41:47.789296 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.789276    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"]
Jul 02 18:41:47.790176 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.790162    2575 scope.go:117] "RemoveContainer" containerID="c3084f09f4d5dbdba1bd75e469e7d52706db52e26cc8613b1d19014fc282bd9b"
Jul 02 18:41:47.795529 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.795510    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-m4sfr"]
Jul 02 18:41:47.797389 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.797376    2575 scope.go:117] "RemoveContainer" containerID="bcf8c385226a2d935df8395d7de5d056ea7c1d5a42fc6ea5614948bf215839be"
Jul 02 18:41:47.797680 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:41:47.797662    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf8c385226a2d935df8395d7de5d056ea7c1d5a42fc6ea5614948bf215839be\": container with ID starting with bcf8c385226a2d935df8395d7de5d056ea7c1d5a42fc6ea5614948bf215839be not found: ID does not exist" containerID="bcf8c385226a2d935df8395d7de5d056ea7c1d5a42fc6ea5614948bf215839be"
Jul 02 18:41:47.797749 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.797685    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf8c385226a2d935df8395d7de5d056ea7c1d5a42fc6ea5614948bf215839be"} err="failed to get container status \"bcf8c385226a2d935df8395d7de5d056ea7c1d5a42fc6ea5614948bf215839be\": rpc error: code = NotFound desc = could not find container \"bcf8c385226a2d935df8395d7de5d056ea7c1d5a42fc6ea5614948bf215839be\": container with ID starting with bcf8c385226a2d935df8395d7de5d056ea7c1d5a42fc6ea5614948bf215839be not found: ID does not exist"
Jul 02 18:41:47.797749 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.797702    2575 scope.go:117] "RemoveContainer" containerID="69bb3c286363b8b7222474e2f69ff7ddad4d5d205b41e1e6efbf641b1587fde5"
Jul 02 18:41:47.797927 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:41:47.797912    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69bb3c286363b8b7222474e2f69ff7ddad4d5d205b41e1e6efbf641b1587fde5\": container with ID starting with 69bb3c286363b8b7222474e2f69ff7ddad4d5d205b41e1e6efbf641b1587fde5 not found: ID does not exist" containerID="69bb3c286363b8b7222474e2f69ff7ddad4d5d205b41e1e6efbf641b1587fde5"
Jul 02 18:41:47.797967 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.797931    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69bb3c286363b8b7222474e2f69ff7ddad4d5d205b41e1e6efbf641b1587fde5"} err="failed to get container status \"69bb3c286363b8b7222474e2f69ff7ddad4d5d205b41e1e6efbf641b1587fde5\": rpc error: code = NotFound desc = could not find container \"69bb3c286363b8b7222474e2f69ff7ddad4d5d205b41e1e6efbf641b1587fde5\": container with ID starting with 69bb3c286363b8b7222474e2f69ff7ddad4d5d205b41e1e6efbf641b1587fde5 not found: ID does not exist"
Jul 02 18:41:47.797967 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.797943    2575 scope.go:117] "RemoveContainer" containerID="c3084f09f4d5dbdba1bd75e469e7d52706db52e26cc8613b1d19014fc282bd9b"
Jul 02 18:41:47.798137 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:41:47.798118    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3084f09f4d5dbdba1bd75e469e7d52706db52e26cc8613b1d19014fc282bd9b\": container with ID starting with c3084f09f4d5dbdba1bd75e469e7d52706db52e26cc8613b1d19014fc282bd9b not found: ID does not exist" containerID="c3084f09f4d5dbdba1bd75e469e7d52706db52e26cc8613b1d19014fc282bd9b"
Jul 02 18:41:47.798181 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:47.798140    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3084f09f4d5dbdba1bd75e469e7d52706db52e26cc8613b1d19014fc282bd9b"} err="failed to get container status \"c3084f09f4d5dbdba1bd75e469e7d52706db52e26cc8613b1d19014fc282bd9b\": rpc error: code = NotFound desc = could not find container \"c3084f09f4d5dbdba1bd75e469e7d52706db52e26cc8613b1d19014fc282bd9b\": container with ID starting with c3084f09f4d5dbdba1bd75e469e7d52706db52e26cc8613b1d19014fc282bd9b not found: ID does not exist"
Jul 02 18:41:48.487727 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:48.487694    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" path="/var/lib/kubelet/pods/1b12d2ba-3425-4dc3-b91d-e709f8436b48/volumes"
Jul 02 18:41:48.488177 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:48.488162    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" path="/var/lib/kubelet/pods/43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97/volumes"
Jul 02 18:41:50.743391 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:50.743365    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:41:50.743880 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:50.743751    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 02 18:41:51.940888 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:51.940848    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:41:53.016385 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:41:53.016341    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:42:00.744101 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:42:00.744012    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 02 18:42:10.744355 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:42:10.744310    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 02 18:42:20.743848 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:42:20.743806    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 02 18:42:30.744337 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:42:30.744309    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:42:39.265399 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:42:39.265362    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:42:39.324344 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:42:39.324310    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:42:41.846365 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:42:41.846330    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:42:42.458584 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:42:42.458543    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:45:08.393084 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:45:08.393056    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 18:45:08.393565 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:45:08.393065    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 18:45:08.401513 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:45:08.401485    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 18:45:08.401681 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:45:08.401486    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 18:45:43.035879 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:45:43.035840    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:47:43.376502 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:47:43.376464    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:49:02.815595 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:49:02.815558    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:05.079769 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:05.079710    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:08.419589 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:08.419559    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 18:50:08.420634 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:08.420611    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 18:50:08.427616 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:08.427583    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 18:50:08.428382 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:08.428360    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 18:50:16.207062 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.207023    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:16.736032 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.735994    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:16.851009 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.850976    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:16.933840 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.933807    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:16.967318 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967289    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"]
Jul 02 18:50:16.967601 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967589    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kserve-container"
Jul 02 18:50:16.967644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967604    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kserve-container"
Jul 02 18:50:16.967644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967620    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kube-rbac-proxy"
Jul 02 18:50:16.967644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967626    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kube-rbac-proxy"
Jul 02 18:50:16.967644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967637    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="storage-initializer"
Jul 02 18:50:16.967644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967642    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="storage-initializer"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967649    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kube-rbac-proxy"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967655    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kube-rbac-proxy"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967661    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="storage-initializer"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967667    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="storage-initializer"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967676    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="storage-initializer"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967682    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="storage-initializer"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967699    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kserve-container"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967705    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kserve-container"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967713    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kserve-container"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967718    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kserve-container"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967728    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kube-rbac-proxy"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967733    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kube-rbac-proxy"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967787    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kserve-container"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967794    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="43aa4427-6e7b-40ba-9e27-6a4fa1d3fe97" containerName="kube-rbac-proxy"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967800    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kserve-container"
Jul 02 18:50:16.967801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967805    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kube-rbac-proxy"
Jul 02 18:50:16.968293 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967810    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b12d2ba-3425-4dc3-b91d-e709f8436b48" containerName="kube-rbac-proxy"
Jul 02 18:50:16.968293 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.967815    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a245f3bd-e1b2-4adb-a2d2-f38e34b31858" containerName="kserve-container"
Jul 02 18:50:16.970698 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.970683    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:16.973550 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.973526    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e22cf-predictor-serving-cert\""
Jul 02 18:50:16.973668 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.973587    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e22cf-kube-rbac-proxy-sar-config\""
Jul 02 18:50:16.980837 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:16.980818    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"]
Jul 02 18:50:17.014093 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.014009    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:17.021205 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.021162    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"]
Jul 02 18:50:17.021430 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.021404    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" podUID="d2698acf-ea23-4665-925e-de1859ba6370" containerName="kserve-container" containerID="cri-o://089bae17bf8a8dac0376ae3a0e1ec586bf5adeb0a702cdd873ae290023fca012" gracePeriod=30
Jul 02 18:50:17.021506 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.021427    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" podUID="d2698acf-ea23-4665-925e-de1859ba6370" containerName="kube-rbac-proxy" containerID="cri-o://3c2bff469783c06f37fe0229701463574d5f4076ed7eadb30b3b913c0abbd699" gracePeriod=30
Jul 02 18:50:17.043889 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.043864    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65e808b-dd46-48a8-a950-a5f65fd8c002-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw\" (UID: \"a65e808b-dd46-48a8-a950-a5f65fd8c002\") " pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:17.044030 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.043896    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g28b5\" (UniqueName: \"kubernetes.io/projected/a65e808b-dd46-48a8-a950-a5f65fd8c002-kube-api-access-g28b5\") pod \"error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw\" (UID: \"a65e808b-dd46-48a8-a950-a5f65fd8c002\") " pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:17.044030 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.043968    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65e808b-dd46-48a8-a950-a5f65fd8c002-proxy-tls\") pod \"error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw\" (UID: \"a65e808b-dd46-48a8-a950-a5f65fd8c002\") " pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:17.114740 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.114704    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:17.145026 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.144999    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65e808b-dd46-48a8-a950-a5f65fd8c002-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw\" (UID: \"a65e808b-dd46-48a8-a950-a5f65fd8c002\") " pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:17.145210 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.145046    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28b5\" (UniqueName: \"kubernetes.io/projected/a65e808b-dd46-48a8-a950-a5f65fd8c002-kube-api-access-g28b5\") pod \"error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw\" (UID: \"a65e808b-dd46-48a8-a950-a5f65fd8c002\") " pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:17.145210 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.145085    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65e808b-dd46-48a8-a950-a5f65fd8c002-proxy-tls\") pod \"error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw\" (UID: \"a65e808b-dd46-48a8-a950-a5f65fd8c002\") " pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:17.145331 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:50:17.145236    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-serving-cert: secret "error-404-isvc-e22cf-predictor-serving-cert" not found
Jul 02 18:50:17.145331 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:50:17.145293    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65e808b-dd46-48a8-a950-a5f65fd8c002-proxy-tls podName:a65e808b-dd46-48a8-a950-a5f65fd8c002 nodeName:}" failed. No retries permitted until 2026-07-02 18:50:17.645277711 +0000 UTC m=+1209.714905218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a65e808b-dd46-48a8-a950-a5f65fd8c002-proxy-tls") pod "error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" (UID: "a65e808b-dd46-48a8-a950-a5f65fd8c002") : secret "error-404-isvc-e22cf-predictor-serving-cert" not found
Jul 02 18:50:17.145654 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.145633    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65e808b-dd46-48a8-a950-a5f65fd8c002-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw\" (UID: \"a65e808b-dd46-48a8-a950-a5f65fd8c002\") " pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:17.156312 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.156291    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28b5\" (UniqueName: \"kubernetes.io/projected/a65e808b-dd46-48a8-a950-a5f65fd8c002-kube-api-access-g28b5\") pod \"error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw\" (UID: \"a65e808b-dd46-48a8-a950-a5f65fd8c002\") " pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:17.357352 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.357320    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:17.651425 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.651341    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65e808b-dd46-48a8-a950-a5f65fd8c002-proxy-tls\") pod \"error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw\" (UID: \"a65e808b-dd46-48a8-a950-a5f65fd8c002\") " pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:17.653691 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.653667    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65e808b-dd46-48a8-a950-a5f65fd8c002-proxy-tls\") pod \"error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw\" (UID: \"a65e808b-dd46-48a8-a950-a5f65fd8c002\") " pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:17.752836 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.752805    2575 generic.go:356] "Generic (PLEG): container finished" podID="d2698acf-ea23-4665-925e-de1859ba6370" containerID="3c2bff469783c06f37fe0229701463574d5f4076ed7eadb30b3b913c0abbd699" exitCode=2
Jul 02 18:50:17.752999 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.752882    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" event={"ID":"d2698acf-ea23-4665-925e-de1859ba6370","Type":"ContainerDied","Data":"3c2bff469783c06f37fe0229701463574d5f4076ed7eadb30b3b913c0abbd699"}
Jul 02 18:50:17.881271 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:17.881240    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:18.009684 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:18.009649    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"]
Jul 02 18:50:18.011504 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:50:18.011475    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65e808b_dd46_48a8_a950_a5f65fd8c002.slice/crio-acdadf3d499c3da3b700f75122fa31f6848cdd5ce71c9939c6c075d0de2684e1 WatchSource:0}: Error finding container acdadf3d499c3da3b700f75122fa31f6848cdd5ce71c9939c6c075d0de2684e1: Status 404 returned error can't find the container with id acdadf3d499c3da3b700f75122fa31f6848cdd5ce71c9939c6c075d0de2684e1
Jul 02 18:50:18.758678 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:18.758643    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" event={"ID":"a65e808b-dd46-48a8-a950-a5f65fd8c002","Type":"ContainerStarted","Data":"7a1ef362fbf6e7881c6e89142d4f6899859b1136be05e15c4ae5304e428cc4b4"}
Jul 02 18:50:18.759153 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:18.758702    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" event={"ID":"a65e808b-dd46-48a8-a950-a5f65fd8c002","Type":"ContainerStarted","Data":"53e6401acbbf76d81c13c2800bedee518cc2a9aa9e1cd033945f0f18973bca62"}
Jul 02 18:50:18.759153 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:18.758723    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" event={"ID":"a65e808b-dd46-48a8-a950-a5f65fd8c002","Type":"ContainerStarted","Data":"acdadf3d499c3da3b700f75122fa31f6848cdd5ce71c9939c6c075d0de2684e1"}
Jul 02 18:50:18.759153 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:18.758756    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:18.777777 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:18.777732    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" podStartSLOduration=2.777719802 podStartE2EDuration="2.777719802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:50:18.775781666 +0000 UTC m=+1210.845409230" watchObservedRunningTime="2026-07-02 18:50:18.777719802 +0000 UTC m=+1210.847347331"
Jul 02 18:50:19.762374 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:19.762341    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:19.763661 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:19.763634    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:50:20.371523 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.371500    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:50:20.473651 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.473568    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbtxd\" (UniqueName: \"kubernetes.io/projected/d2698acf-ea23-4665-925e-de1859ba6370-kube-api-access-xbtxd\") pod \"d2698acf-ea23-4665-925e-de1859ba6370\" (UID: \"d2698acf-ea23-4665-925e-de1859ba6370\") "
Jul 02 18:50:20.473651 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.473614    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2698acf-ea23-4665-925e-de1859ba6370-kube-rbac-proxy-sar-config\") pod \"d2698acf-ea23-4665-925e-de1859ba6370\" (UID: \"d2698acf-ea23-4665-925e-de1859ba6370\") "
Jul 02 18:50:20.473651 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.473648    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2698acf-ea23-4665-925e-de1859ba6370-proxy-tls\") pod \"d2698acf-ea23-4665-925e-de1859ba6370\" (UID: \"d2698acf-ea23-4665-925e-de1859ba6370\") "
Jul 02 18:50:20.473977 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.473955    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2698acf-ea23-4665-925e-de1859ba6370-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2698acf-ea23-4665-925e-de1859ba6370" (UID: "d2698acf-ea23-4665-925e-de1859ba6370"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:50:20.475701 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.475678    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2698acf-ea23-4665-925e-de1859ba6370-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2698acf-ea23-4665-925e-de1859ba6370" (UID: "d2698acf-ea23-4665-925e-de1859ba6370"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:50:20.475814 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.475756    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2698acf-ea23-4665-925e-de1859ba6370-kube-api-access-xbtxd" (OuterVolumeSpecName: "kube-api-access-xbtxd") pod "d2698acf-ea23-4665-925e-de1859ba6370" (UID: "d2698acf-ea23-4665-925e-de1859ba6370"). InnerVolumeSpecName "kube-api-access-xbtxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:50:20.574801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.574766    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbtxd\" (UniqueName: \"kubernetes.io/projected/d2698acf-ea23-4665-925e-de1859ba6370-kube-api-access-xbtxd\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:50:20.574801 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.574796    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2698acf-ea23-4665-925e-de1859ba6370-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:50:20.575001 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.574809    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2698acf-ea23-4665-925e-de1859ba6370-proxy-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:50:20.768100 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.768010    2575 generic.go:356] "Generic (PLEG): container finished" podID="d2698acf-ea23-4665-925e-de1859ba6370" containerID="089bae17bf8a8dac0376ae3a0e1ec586bf5adeb0a702cdd873ae290023fca012" exitCode=0
Jul 02 18:50:20.768100 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.768062    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"
Jul 02 18:50:20.768100 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.768096    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" event={"ID":"d2698acf-ea23-4665-925e-de1859ba6370","Type":"ContainerDied","Data":"089bae17bf8a8dac0376ae3a0e1ec586bf5adeb0a702cdd873ae290023fca012"}
Jul 02 18:50:20.768713 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.768171    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn" event={"ID":"d2698acf-ea23-4665-925e-de1859ba6370","Type":"ContainerDied","Data":"76cb855194a94629a6070cfcb0ee4b793251874839746f9aae691d0d60541282"}
Jul 02 18:50:20.768713 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.768234    2575 scope.go:117] "RemoveContainer" containerID="3c2bff469783c06f37fe0229701463574d5f4076ed7eadb30b3b913c0abbd699"
Jul 02 18:50:20.768713 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.768622    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:50:20.776283 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.776265    2575 scope.go:117] "RemoveContainer" containerID="089bae17bf8a8dac0376ae3a0e1ec586bf5adeb0a702cdd873ae290023fca012"
Jul 02 18:50:20.783370 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.783318    2575 scope.go:117] "RemoveContainer" containerID="3c2bff469783c06f37fe0229701463574d5f4076ed7eadb30b3b913c0abbd699"
Jul 02 18:50:20.783640 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:50:20.783622    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2bff469783c06f37fe0229701463574d5f4076ed7eadb30b3b913c0abbd699\": container with ID starting with 3c2bff469783c06f37fe0229701463574d5f4076ed7eadb30b3b913c0abbd699 not found: ID does not exist" containerID="3c2bff469783c06f37fe0229701463574d5f4076ed7eadb30b3b913c0abbd699"
Jul 02 18:50:20.783705 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.783649    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2bff469783c06f37fe0229701463574d5f4076ed7eadb30b3b913c0abbd699"} err="failed to get container status \"3c2bff469783c06f37fe0229701463574d5f4076ed7eadb30b3b913c0abbd699\": rpc error: code = NotFound desc = could not find container \"3c2bff469783c06f37fe0229701463574d5f4076ed7eadb30b3b913c0abbd699\": container with ID starting with 3c2bff469783c06f37fe0229701463574d5f4076ed7eadb30b3b913c0abbd699 not found: ID does not exist"
Jul 02 18:50:20.783705 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.783665    2575 scope.go:117] "RemoveContainer" containerID="089bae17bf8a8dac0376ae3a0e1ec586bf5adeb0a702cdd873ae290023fca012"
Jul 02 18:50:20.783939 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:50:20.783922    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089bae17bf8a8dac0376ae3a0e1ec586bf5adeb0a702cdd873ae290023fca012\": container with ID starting with 089bae17bf8a8dac0376ae3a0e1ec586bf5adeb0a702cdd873ae290023fca012 not found: ID does not exist" containerID="089bae17bf8a8dac0376ae3a0e1ec586bf5adeb0a702cdd873ae290023fca012"
Jul 02 18:50:20.783978 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.783944    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089bae17bf8a8dac0376ae3a0e1ec586bf5adeb0a702cdd873ae290023fca012"} err="failed to get container status \"089bae17bf8a8dac0376ae3a0e1ec586bf5adeb0a702cdd873ae290023fca012\": rpc error: code = NotFound desc = could not find container \"089bae17bf8a8dac0376ae3a0e1ec586bf5adeb0a702cdd873ae290023fca012\": container with ID starting with 089bae17bf8a8dac0376ae3a0e1ec586bf5adeb0a702cdd873ae290023fca012 not found: ID does not exist"
Jul 02 18:50:20.785121 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.785102    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"]
Jul 02 18:50:20.786433 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:20.786414    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a5373-predictor-7b4cdc944b-lq8zn"]
Jul 02 18:50:22.487754 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:22.487720    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2698acf-ea23-4665-925e-de1859ba6370" path="/var/lib/kubelet/pods/d2698acf-ea23-4665-925e-de1859ba6370/volumes"
Jul 02 18:50:25.773725 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:25.773698    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:50:25.774224 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:25.774183    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:50:35.774876 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:35.774840    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:50:45.775110 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:45.775071    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:50:55.775095 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:55.775048    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:50:56.602851 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.602784    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:56.711557 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.711522    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:56.808466 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.808427    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:56.810086 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.810060    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"]
Jul 02 18:50:56.810478 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.810461    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2698acf-ea23-4665-925e-de1859ba6370" containerName="kserve-container"
Jul 02 18:50:56.810560 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.810480    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2698acf-ea23-4665-925e-de1859ba6370" containerName="kserve-container"
Jul 02 18:50:56.810560 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.810492    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2698acf-ea23-4665-925e-de1859ba6370" containerName="kube-rbac-proxy"
Jul 02 18:50:56.810560 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.810502    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2698acf-ea23-4665-925e-de1859ba6370" containerName="kube-rbac-proxy"
Jul 02 18:50:56.810716 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.810579    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2698acf-ea23-4665-925e-de1859ba6370" containerName="kube-rbac-proxy"
Jul 02 18:50:56.810716 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.810593    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2698acf-ea23-4665-925e-de1859ba6370" containerName="kserve-container"
Jul 02 18:50:56.813678 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.813658    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:50:56.816438 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.816414    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-94e8c-predictor-serving-cert\""
Jul 02 18:50:56.816556 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.816445    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-94e8c-kube-rbac-proxy-sar-config\""
Jul 02 18:50:56.823828 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.823800    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"]
Jul 02 18:50:56.861481 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.861397    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"]
Jul 02 18:50:56.861786 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.861751    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" containerName="kserve-container" containerID="cri-o://d8e6b2fcbf561946d8c0e22874aa91c4884238f8ed0cdaa72ffdc7eb8e324351" gracePeriod=30
Jul 02 18:50:56.862725 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.861978    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" containerName="kube-rbac-proxy" containerID="cri-o://71c387d4a80f68e49d1290b4421d3494a5c587849cef51784c9b4ee1ddc563b5" gracePeriod=30
Jul 02 18:50:56.869534 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.869502    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:56.878875 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.878849    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh78c\" (UniqueName: \"kubernetes.io/projected/ad7220bc-f3d6-40c9-830c-35024c9c3b76-kube-api-access-gh78c\") pod \"error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf\" (UID: \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\") " pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:50:56.878970 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.878896    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad7220bc-f3d6-40c9-830c-35024c9c3b76-proxy-tls\") pod \"error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf\" (UID: \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\") " pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:50:56.878970 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.878914    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad7220bc-f3d6-40c9-830c-35024c9c3b76-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf\" (UID: \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\") " pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:50:56.968310 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.968274    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:56.979556 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.979531    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh78c\" (UniqueName: \"kubernetes.io/projected/ad7220bc-f3d6-40c9-830c-35024c9c3b76-kube-api-access-gh78c\") pod \"error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf\" (UID: \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\") " pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:50:56.979682 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.979571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad7220bc-f3d6-40c9-830c-35024c9c3b76-proxy-tls\") pod \"error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf\" (UID: \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\") " pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:50:56.979682 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.979590    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad7220bc-f3d6-40c9-830c-35024c9c3b76-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf\" (UID: \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\") " pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:50:56.979763 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:50:56.979696    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-serving-cert: secret "error-404-isvc-94e8c-predictor-serving-cert" not found
Jul 02 18:50:56.979813 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:50:56.979787    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad7220bc-f3d6-40c9-830c-35024c9c3b76-proxy-tls podName:ad7220bc-f3d6-40c9-830c-35024c9c3b76 nodeName:}" failed. No retries permitted until 2026-07-02 18:50:57.479756711 +0000 UTC m=+1249.549384218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad7220bc-f3d6-40c9-830c-35024c9c3b76-proxy-tls") pod "error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" (UID: "ad7220bc-f3d6-40c9-830c-35024c9c3b76") : secret "error-404-isvc-94e8c-predictor-serving-cert" not found
Jul 02 18:50:56.980174 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.980149    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad7220bc-f3d6-40c9-830c-35024c9c3b76-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf\" (UID: \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\") " pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:50:56.987745 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:56.987727    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh78c\" (UniqueName: \"kubernetes.io/projected/ad7220bc-f3d6-40c9-830c-35024c9c3b76-kube-api-access-gh78c\") pod \"error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf\" (UID: \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\") " pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:50:57.132887 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:57.132800    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:57.233582 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:57.233539    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:50:57.484092 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:57.484005    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad7220bc-f3d6-40c9-830c-35024c9c3b76-proxy-tls\") pod \"error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf\" (UID: \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\") " pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:50:57.486461 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:57.486439    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad7220bc-f3d6-40c9-830c-35024c9c3b76-proxy-tls\") pod \"error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf\" (UID: \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\") " pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:50:57.725629 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:57.725585    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:50:57.855246 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:57.855226    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"]
Jul 02 18:50:57.856851 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:50:57.856828    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7220bc_f3d6_40c9_830c_35024c9c3b76.slice/crio-6147edcfccdbba3cf8911cd3cc6e89da9aed463ef3c997522526e39bbe7dcc5f WatchSource:0}: Error finding container 6147edcfccdbba3cf8911cd3cc6e89da9aed463ef3c997522526e39bbe7dcc5f: Status 404 returned error can't find the container with id 6147edcfccdbba3cf8911cd3cc6e89da9aed463ef3c997522526e39bbe7dcc5f
Jul 02 18:50:57.913222 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:57.913176    2575 generic.go:356] "Generic (PLEG): container finished" podID="fefe989b-4300-4eb7-bb66-816c42550acc" containerID="71c387d4a80f68e49d1290b4421d3494a5c587849cef51784c9b4ee1ddc563b5" exitCode=2
Jul 02 18:50:57.913326 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:57.913222    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" event={"ID":"fefe989b-4300-4eb7-bb66-816c42550acc","Type":"ContainerDied","Data":"71c387d4a80f68e49d1290b4421d3494a5c587849cef51784c9b4ee1ddc563b5"}
Jul 02 18:50:57.914797 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:57.914775    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" event={"ID":"ad7220bc-f3d6-40c9-830c-35024c9c3b76","Type":"ContainerStarted","Data":"6147edcfccdbba3cf8911cd3cc6e89da9aed463ef3c997522526e39bbe7dcc5f"}
Jul 02 18:50:58.920010 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:58.919975    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" event={"ID":"ad7220bc-f3d6-40c9-830c-35024c9c3b76","Type":"ContainerStarted","Data":"228c53648c78133a17d7bdbd34cf0a48476ae1ea4f758c503c5bec20eeb118b9"}
Jul 02 18:50:58.920453 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:58.920040    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" event={"ID":"ad7220bc-f3d6-40c9-830c-35024c9c3b76","Type":"ContainerStarted","Data":"1c2273ea57feadd7495cd98488f07c60cbae5f1af452f753f82b400c12a40cc7"}
Jul 02 18:50:58.920453 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:58.920118    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:50:58.937103 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:58.937047    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" podStartSLOduration=2.937032518 podStartE2EDuration="2.937032518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:50:58.936788364 +0000 UTC m=+1251.006415916" watchObservedRunningTime="2026-07-02 18:50:58.937032518 +0000 UTC m=+1251.006660045"
Jul 02 18:50:59.923894 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:59.923854    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:50:59.925181 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:50:59.925150    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 18:51:00.613547 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.613517    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:51:00.708816 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.708775    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fefe989b-4300-4eb7-bb66-816c42550acc-kube-rbac-proxy-sar-config\") pod \"fefe989b-4300-4eb7-bb66-816c42550acc\" (UID: \"fefe989b-4300-4eb7-bb66-816c42550acc\") "
Jul 02 18:51:00.709031 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.708831    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvqkp\" (UniqueName: \"kubernetes.io/projected/fefe989b-4300-4eb7-bb66-816c42550acc-kube-api-access-lvqkp\") pod \"fefe989b-4300-4eb7-bb66-816c42550acc\" (UID: \"fefe989b-4300-4eb7-bb66-816c42550acc\") "
Jul 02 18:51:00.709031 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.708891    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fefe989b-4300-4eb7-bb66-816c42550acc-proxy-tls\") pod \"fefe989b-4300-4eb7-bb66-816c42550acc\" (UID: \"fefe989b-4300-4eb7-bb66-816c42550acc\") "
Jul 02 18:51:00.709265 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.709232    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefe989b-4300-4eb7-bb66-816c42550acc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fefe989b-4300-4eb7-bb66-816c42550acc" (UID: "fefe989b-4300-4eb7-bb66-816c42550acc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:51:00.711076 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.711051    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefe989b-4300-4eb7-bb66-816c42550acc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fefe989b-4300-4eb7-bb66-816c42550acc" (UID: "fefe989b-4300-4eb7-bb66-816c42550acc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:51:00.711235 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.711060    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefe989b-4300-4eb7-bb66-816c42550acc-kube-api-access-lvqkp" (OuterVolumeSpecName: "kube-api-access-lvqkp") pod "fefe989b-4300-4eb7-bb66-816c42550acc" (UID: "fefe989b-4300-4eb7-bb66-816c42550acc"). InnerVolumeSpecName "kube-api-access-lvqkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:51:00.810462 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.810436    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fefe989b-4300-4eb7-bb66-816c42550acc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:51:00.810462 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.810463    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvqkp\" (UniqueName: \"kubernetes.io/projected/fefe989b-4300-4eb7-bb66-816c42550acc-kube-api-access-lvqkp\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:51:00.810663 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.810475    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fefe989b-4300-4eb7-bb66-816c42550acc-proxy-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:51:00.929184 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.929145    2575 generic.go:356] "Generic (PLEG): container finished" podID="fefe989b-4300-4eb7-bb66-816c42550acc" containerID="d8e6b2fcbf561946d8c0e22874aa91c4884238f8ed0cdaa72ffdc7eb8e324351" exitCode=0
Jul 02 18:51:00.929677 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.929213    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"
Jul 02 18:51:00.929677 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.929222    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" event={"ID":"fefe989b-4300-4eb7-bb66-816c42550acc","Type":"ContainerDied","Data":"d8e6b2fcbf561946d8c0e22874aa91c4884238f8ed0cdaa72ffdc7eb8e324351"}
Jul 02 18:51:00.929677 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.929280    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl" event={"ID":"fefe989b-4300-4eb7-bb66-816c42550acc","Type":"ContainerDied","Data":"4f15c033f23ac84f9b202035d5e7a48bda9877e1555e962325258d6a61fcd5e7"}
Jul 02 18:51:00.929677 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.929308    2575 scope.go:117] "RemoveContainer" containerID="71c387d4a80f68e49d1290b4421d3494a5c587849cef51784c9b4ee1ddc563b5"
Jul 02 18:51:00.929904 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.929858    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 18:51:00.938662 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.938646    2575 scope.go:117] "RemoveContainer" containerID="d8e6b2fcbf561946d8c0e22874aa91c4884238f8ed0cdaa72ffdc7eb8e324351"
Jul 02 18:51:00.948123 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.948104    2575 scope.go:117] "RemoveContainer" containerID="71c387d4a80f68e49d1290b4421d3494a5c587849cef51784c9b4ee1ddc563b5"
Jul 02 18:51:00.948412 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:51:00.948390    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c387d4a80f68e49d1290b4421d3494a5c587849cef51784c9b4ee1ddc563b5\": container with ID starting with 71c387d4a80f68e49d1290b4421d3494a5c587849cef51784c9b4ee1ddc563b5 not found: ID does not exist" containerID="71c387d4a80f68e49d1290b4421d3494a5c587849cef51784c9b4ee1ddc563b5"
Jul 02 18:51:00.948470 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.948423    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c387d4a80f68e49d1290b4421d3494a5c587849cef51784c9b4ee1ddc563b5"} err="failed to get container status \"71c387d4a80f68e49d1290b4421d3494a5c587849cef51784c9b4ee1ddc563b5\": rpc error: code = NotFound desc = could not find container \"71c387d4a80f68e49d1290b4421d3494a5c587849cef51784c9b4ee1ddc563b5\": container with ID starting with 71c387d4a80f68e49d1290b4421d3494a5c587849cef51784c9b4ee1ddc563b5 not found: ID does not exist"
Jul 02 18:51:00.948470 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.948441    2575 scope.go:117] "RemoveContainer" containerID="d8e6b2fcbf561946d8c0e22874aa91c4884238f8ed0cdaa72ffdc7eb8e324351"
Jul 02 18:51:00.948815 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:51:00.948746    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e6b2fcbf561946d8c0e22874aa91c4884238f8ed0cdaa72ffdc7eb8e324351\": container with ID starting with d8e6b2fcbf561946d8c0e22874aa91c4884238f8ed0cdaa72ffdc7eb8e324351 not found: ID does not exist" containerID="d8e6b2fcbf561946d8c0e22874aa91c4884238f8ed0cdaa72ffdc7eb8e324351"
Jul 02 18:51:00.948815 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.948779    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e6b2fcbf561946d8c0e22874aa91c4884238f8ed0cdaa72ffdc7eb8e324351"} err="failed to get container status \"d8e6b2fcbf561946d8c0e22874aa91c4884238f8ed0cdaa72ffdc7eb8e324351\": rpc error: code = NotFound desc = could not find container \"d8e6b2fcbf561946d8c0e22874aa91c4884238f8ed0cdaa72ffdc7eb8e324351\": container with ID starting with d8e6b2fcbf561946d8c0e22874aa91c4884238f8ed0cdaa72ffdc7eb8e324351 not found: ID does not exist"
Jul 02 18:51:00.950835 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.950805    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"]
Jul 02 18:51:00.955893 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:00.955861    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8b76-predictor-756c9998f6-2lnkl"]
Jul 02 18:51:02.488618 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:02.488584    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" path="/var/lib/kubelet/pods/fefe989b-4300-4eb7-bb66-816c42550acc/volumes"
Jul 02 18:51:05.775178 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:05.775148    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:51:05.935365 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:05.935335    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:51:05.935926 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:05.935890    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 18:51:15.936111 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:15.936069    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 18:51:16.945644 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:16.945610    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:51:17.452252 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:17.452220    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:51:25.935990 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:25.935951    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 18:51:27.028871 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.028774    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:51:27.141914 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.141881    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:51:27.224469 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.224432    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"]
Jul 02 18:51:27.224759 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.224731    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerName="kserve-container" containerID="cri-o://53e6401acbbf76d81c13c2800bedee518cc2a9aa9e1cd033945f0f18973bca62" gracePeriod=30
Jul 02 18:51:27.224840 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.224760    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerName="kube-rbac-proxy" containerID="cri-o://7a1ef362fbf6e7881c6e89142d4f6899859b1136be05e15c4ae5304e428cc4b4" gracePeriod=30
Jul 02 18:51:27.230898 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.229855    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:51:27.253270 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.253243    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"]
Jul 02 18:51:27.253550 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.253538    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" containerName="kube-rbac-proxy"
Jul 02 18:51:27.253592 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.253551    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" containerName="kube-rbac-proxy"
Jul 02 18:51:27.253592 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.253566    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" containerName="kserve-container"
Jul 02 18:51:27.253592 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.253572    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" containerName="kserve-container"
Jul 02 18:51:27.253684 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.253621    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" containerName="kube-rbac-proxy"
Jul 02 18:51:27.253684 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.253631    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fefe989b-4300-4eb7-bb66-816c42550acc" containerName="kserve-container"
Jul 02 18:51:27.257886 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.257870    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:27.260431 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.260409    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7acc8-predictor-serving-cert\""
Jul 02 18:51:27.260518 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.260431    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7acc8-kube-rbac-proxy-sar-config\""
Jul 02 18:51:27.271052 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.267478    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"]
Jul 02 18:51:27.291080 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.290998    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:51:27.331013 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.330984    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147ea518-52f2-41bf-8426-7f55a55d04d5-proxy-tls\") pod \"error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l\" (UID: \"147ea518-52f2-41bf-8426-7f55a55d04d5\") " pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:27.331187 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.331044    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjrc4\" (UniqueName: \"kubernetes.io/projected/147ea518-52f2-41bf-8426-7f55a55d04d5-kube-api-access-kjrc4\") pod \"error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l\" (UID: \"147ea518-52f2-41bf-8426-7f55a55d04d5\") " pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:27.331187 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.331129    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147ea518-52f2-41bf-8426-7f55a55d04d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l\" (UID: \"147ea518-52f2-41bf-8426-7f55a55d04d5\") " pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:27.431936 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.431885    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147ea518-52f2-41bf-8426-7f55a55d04d5-proxy-tls\") pod \"error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l\" (UID: \"147ea518-52f2-41bf-8426-7f55a55d04d5\") " pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:27.432145 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.431977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjrc4\" (UniqueName: \"kubernetes.io/projected/147ea518-52f2-41bf-8426-7f55a55d04d5-kube-api-access-kjrc4\") pod \"error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l\" (UID: \"147ea518-52f2-41bf-8426-7f55a55d04d5\") " pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:27.432145 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.432030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147ea518-52f2-41bf-8426-7f55a55d04d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l\" (UID: \"147ea518-52f2-41bf-8426-7f55a55d04d5\") " pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:27.432145 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:51:27.432038    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-serving-cert: secret "error-404-isvc-7acc8-predictor-serving-cert" not found
Jul 02 18:51:27.432145 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:51:27.432124    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/147ea518-52f2-41bf-8426-7f55a55d04d5-proxy-tls podName:147ea518-52f2-41bf-8426-7f55a55d04d5 nodeName:}" failed. No retries permitted until 2026-07-02 18:51:27.932103147 +0000 UTC m=+1280.001730653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/147ea518-52f2-41bf-8426-7f55a55d04d5-proxy-tls") pod "error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" (UID: "147ea518-52f2-41bf-8426-7f55a55d04d5") : secret "error-404-isvc-7acc8-predictor-serving-cert" not found
Jul 02 18:51:27.432741 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.432713    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147ea518-52f2-41bf-8426-7f55a55d04d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l\" (UID: \"147ea518-52f2-41bf-8426-7f55a55d04d5\") " pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:27.440982 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.440950    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjrc4\" (UniqueName: \"kubernetes.io/projected/147ea518-52f2-41bf-8426-7f55a55d04d5-kube-api-access-kjrc4\") pod \"error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l\" (UID: \"147ea518-52f2-41bf-8426-7f55a55d04d5\") " pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:27.531262 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.531214    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:51:27.936028 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.935992    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147ea518-52f2-41bf-8426-7f55a55d04d5-proxy-tls\") pod \"error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l\" (UID: \"147ea518-52f2-41bf-8426-7f55a55d04d5\") " pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:27.938429 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:27.938400    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147ea518-52f2-41bf-8426-7f55a55d04d5-proxy-tls\") pod \"error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l\" (UID: \"147ea518-52f2-41bf-8426-7f55a55d04d5\") " pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:28.040721 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:28.040687    2575 generic.go:356] "Generic (PLEG): container finished" podID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerID="7a1ef362fbf6e7881c6e89142d4f6899859b1136be05e15c4ae5304e428cc4b4" exitCode=2
Jul 02 18:51:28.041083 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:28.040768    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" event={"ID":"a65e808b-dd46-48a8-a950-a5f65fd8c002","Type":"ContainerDied","Data":"7a1ef362fbf6e7881c6e89142d4f6899859b1136be05e15c4ae5304e428cc4b4"}
Jul 02 18:51:28.170958 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:28.170918    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:28.299218 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:28.299176    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"]
Jul 02 18:51:28.301337 ip-10-0-140-68 kubenswrapper[2575]: W0702 18:51:28.301307    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147ea518_52f2_41bf_8426_7f55a55d04d5.slice/crio-f91cecd6e4e6ef8e242d51bd0c561f6ac8646bc5b01c6bac1d7060c99680e036 WatchSource:0}: Error finding container f91cecd6e4e6ef8e242d51bd0c561f6ac8646bc5b01c6bac1d7060c99680e036: Status 404 returned error can't find the container with id f91cecd6e4e6ef8e242d51bd0c561f6ac8646bc5b01c6bac1d7060c99680e036
Jul 02 18:51:29.046579 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:29.046536    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" event={"ID":"147ea518-52f2-41bf-8426-7f55a55d04d5","Type":"ContainerStarted","Data":"ed3964c08b089fcc2259321cb5831632b56a4c3b8a76e90794b29a9aa7669476"}
Jul 02 18:51:29.047057 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:29.046617    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:29.047057 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:29.046639    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" event={"ID":"147ea518-52f2-41bf-8426-7f55a55d04d5","Type":"ContainerStarted","Data":"708f23dad50eb619419dd18f1cf46c2428f2aea67b4e7a99a66ca3d691e780ae"}
Jul 02 18:51:29.047057 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:29.046661    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:29.047057 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:29.046678    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" event={"ID":"147ea518-52f2-41bf-8426-7f55a55d04d5","Type":"ContainerStarted","Data":"f91cecd6e4e6ef8e242d51bd0c561f6ac8646bc5b01c6bac1d7060c99680e036"}
Jul 02 18:51:29.048066 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:29.048038    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 18:51:29.063883 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:29.063840    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" podStartSLOduration=2.063825084 podStartE2EDuration="2.063825084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:51:29.063016351 +0000 UTC m=+1281.132643900" watchObservedRunningTime="2026-07-02 18:51:29.063825084 +0000 UTC m=+1281.133452612"
Jul 02 18:51:30.050669 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:30.050631    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 18:51:30.774500 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:30.774477    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:51:30.861993 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:30.861960    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65e808b-dd46-48a8-a950-a5f65fd8c002-proxy-tls\") pod \"a65e808b-dd46-48a8-a950-a5f65fd8c002\" (UID: \"a65e808b-dd46-48a8-a950-a5f65fd8c002\") "
Jul 02 18:51:30.862166 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:30.862027    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65e808b-dd46-48a8-a950-a5f65fd8c002-kube-rbac-proxy-sar-config\") pod \"a65e808b-dd46-48a8-a950-a5f65fd8c002\" (UID: \"a65e808b-dd46-48a8-a950-a5f65fd8c002\") "
Jul 02 18:51:30.862166 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:30.862052    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g28b5\" (UniqueName: \"kubernetes.io/projected/a65e808b-dd46-48a8-a950-a5f65fd8c002-kube-api-access-g28b5\") pod \"a65e808b-dd46-48a8-a950-a5f65fd8c002\" (UID: \"a65e808b-dd46-48a8-a950-a5f65fd8c002\") "
Jul 02 18:51:30.862416 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:30.862391    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65e808b-dd46-48a8-a950-a5f65fd8c002-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a65e808b-dd46-48a8-a950-a5f65fd8c002" (UID: "a65e808b-dd46-48a8-a950-a5f65fd8c002"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:51:30.864014 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:30.863985    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65e808b-dd46-48a8-a950-a5f65fd8c002-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a65e808b-dd46-48a8-a950-a5f65fd8c002" (UID: "a65e808b-dd46-48a8-a950-a5f65fd8c002"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:51:30.864112 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:30.864049    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65e808b-dd46-48a8-a950-a5f65fd8c002-kube-api-access-g28b5" (OuterVolumeSpecName: "kube-api-access-g28b5") pod "a65e808b-dd46-48a8-a950-a5f65fd8c002" (UID: "a65e808b-dd46-48a8-a950-a5f65fd8c002"). InnerVolumeSpecName "kube-api-access-g28b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:51:30.962752 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:30.962717    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65e808b-dd46-48a8-a950-a5f65fd8c002-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:51:30.962752 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:30.962744    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g28b5\" (UniqueName: \"kubernetes.io/projected/a65e808b-dd46-48a8-a950-a5f65fd8c002-kube-api-access-g28b5\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:51:30.962752 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:30.962756    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65e808b-dd46-48a8-a950-a5f65fd8c002-proxy-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:51:31.055205 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:31.055156    2575 generic.go:356] "Generic (PLEG): container finished" podID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerID="53e6401acbbf76d81c13c2800bedee518cc2a9aa9e1cd033945f0f18973bca62" exitCode=0
Jul 02 18:51:31.055623 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:31.055224    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"
Jul 02 18:51:31.055623 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:31.055235    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" event={"ID":"a65e808b-dd46-48a8-a950-a5f65fd8c002","Type":"ContainerDied","Data":"53e6401acbbf76d81c13c2800bedee518cc2a9aa9e1cd033945f0f18973bca62"}
Jul 02 18:51:31.055623 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:31.055300    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw" event={"ID":"a65e808b-dd46-48a8-a950-a5f65fd8c002","Type":"ContainerDied","Data":"acdadf3d499c3da3b700f75122fa31f6848cdd5ce71c9939c6c075d0de2684e1"}
Jul 02 18:51:31.055623 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:31.055343    2575 scope.go:117] "RemoveContainer" containerID="7a1ef362fbf6e7881c6e89142d4f6899859b1136be05e15c4ae5304e428cc4b4"
Jul 02 18:51:31.063984 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:31.063970    2575 scope.go:117] "RemoveContainer" containerID="53e6401acbbf76d81c13c2800bedee518cc2a9aa9e1cd033945f0f18973bca62"
Jul 02 18:51:31.071045 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:31.071027    2575 scope.go:117] "RemoveContainer" containerID="7a1ef362fbf6e7881c6e89142d4f6899859b1136be05e15c4ae5304e428cc4b4"
Jul 02 18:51:31.071290 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:51:31.071273    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1ef362fbf6e7881c6e89142d4f6899859b1136be05e15c4ae5304e428cc4b4\": container with ID starting with 7a1ef362fbf6e7881c6e89142d4f6899859b1136be05e15c4ae5304e428cc4b4 not found: ID does not exist" containerID="7a1ef362fbf6e7881c6e89142d4f6899859b1136be05e15c4ae5304e428cc4b4"
Jul 02 18:51:31.071335 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:31.071298    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1ef362fbf6e7881c6e89142d4f6899859b1136be05e15c4ae5304e428cc4b4"} err="failed to get container status \"7a1ef362fbf6e7881c6e89142d4f6899859b1136be05e15c4ae5304e428cc4b4\": rpc error: code = NotFound desc = could not find container \"7a1ef362fbf6e7881c6e89142d4f6899859b1136be05e15c4ae5304e428cc4b4\": container with ID starting with 7a1ef362fbf6e7881c6e89142d4f6899859b1136be05e15c4ae5304e428cc4b4 not found: ID does not exist"
Jul 02 18:51:31.071335 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:31.071318    2575 scope.go:117] "RemoveContainer" containerID="53e6401acbbf76d81c13c2800bedee518cc2a9aa9e1cd033945f0f18973bca62"
Jul 02 18:51:31.071521 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:51:31.071505    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e6401acbbf76d81c13c2800bedee518cc2a9aa9e1cd033945f0f18973bca62\": container with ID starting with 53e6401acbbf76d81c13c2800bedee518cc2a9aa9e1cd033945f0f18973bca62 not found: ID does not exist" containerID="53e6401acbbf76d81c13c2800bedee518cc2a9aa9e1cd033945f0f18973bca62"
Jul 02 18:51:31.071572 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:31.071524    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e6401acbbf76d81c13c2800bedee518cc2a9aa9e1cd033945f0f18973bca62"} err="failed to get container status \"53e6401acbbf76d81c13c2800bedee518cc2a9aa9e1cd033945f0f18973bca62\": rpc error: code = NotFound desc = could not find container \"53e6401acbbf76d81c13c2800bedee518cc2a9aa9e1cd033945f0f18973bca62\": container with ID starting with 53e6401acbbf76d81c13c2800bedee518cc2a9aa9e1cd033945f0f18973bca62 not found: ID does not exist"
Jul 02 18:51:31.077121 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:31.077101    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"]
Jul 02 18:51:31.079030 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:31.079011    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e22cf-predictor-77d4dd487b-cdhfw"]
Jul 02 18:51:32.488206 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:32.488167    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" path="/var/lib/kubelet/pods/a65e808b-dd46-48a8-a950-a5f65fd8c002/volumes"
Jul 02 18:51:35.055585 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:35.055557    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:51:35.056014 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:35.055970    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 18:51:35.936223 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:35.936165    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 18:51:45.056332 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:45.056295    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 18:51:45.936839 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:45.936812    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:51:51.314982 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:51.314949    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:51:55.056264 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:55.056223    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 18:51:56.758176 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:56.758137    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:51:56.943859 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:51:56.943813    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:52:04.965302 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:04.965265    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:52:05.015322 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:05.015283    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:52:05.056683 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:05.056639    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 18:52:06.868135 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:06.868107    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:52:06.974582 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:06.974547    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:52:07.076319 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.076259    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:52:07.077211 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.077172    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"]
Jul 02 18:52:07.077517 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.077503    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerName="kube-rbac-proxy"
Jul 02 18:52:07.077517 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.077517    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerName="kube-rbac-proxy"
Jul 02 18:52:07.077610 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.077537    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerName="kserve-container"
Jul 02 18:52:07.077610 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.077542    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerName="kserve-container"
Jul 02 18:52:07.077610 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.077586    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerName="kube-rbac-proxy"
Jul 02 18:52:07.077610 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.077596    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65e808b-dd46-48a8-a950-a5f65fd8c002" containerName="kserve-container"
Jul 02 18:52:07.080853 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.080835    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:07.083514 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.083497    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ae135-kube-rbac-proxy-sar-config\""
Jul 02 18:52:07.083769 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.083749    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ae135-predictor-serving-cert\""
Jul 02 18:52:07.091605 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.091587    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"]
Jul 02 18:52:07.141600 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.141489    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:52:07.142574 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.142550    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"]
Jul 02 18:52:07.142829 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.142804    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerName="kserve-container" containerID="cri-o://1c2273ea57feadd7495cd98488f07c60cbae5f1af452f753f82b400c12a40cc7" gracePeriod=30
Jul 02 18:52:07.142899 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.142832    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerName="kube-rbac-proxy" containerID="cri-o://228c53648c78133a17d7bdbd34cf0a48476ae1ea4f758c503c5bec20eeb118b9" gracePeriod=30
Jul 02 18:52:07.169661 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.169639    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2838a1ec-8335-4af1-8956-fd80ed486104-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ae135-predictor-5dfb866474-tfrnf\" (UID: \"2838a1ec-8335-4af1-8956-fd80ed486104\") " pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:07.169767 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.169667    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2vt7\" (UniqueName: \"kubernetes.io/projected/2838a1ec-8335-4af1-8956-fd80ed486104-kube-api-access-q2vt7\") pod \"error-404-isvc-ae135-predictor-5dfb866474-tfrnf\" (UID: \"2838a1ec-8335-4af1-8956-fd80ed486104\") " pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:07.169767 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.169702    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2838a1ec-8335-4af1-8956-fd80ed486104-proxy-tls\") pod \"error-404-isvc-ae135-predictor-5dfb866474-tfrnf\" (UID: \"2838a1ec-8335-4af1-8956-fd80ed486104\") " pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:07.238734 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.238699    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:52:07.270569 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.270548    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2838a1ec-8335-4af1-8956-fd80ed486104-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ae135-predictor-5dfb866474-tfrnf\" (UID: \"2838a1ec-8335-4af1-8956-fd80ed486104\") " pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:07.270668 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.270579    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2vt7\" (UniqueName: \"kubernetes.io/projected/2838a1ec-8335-4af1-8956-fd80ed486104-kube-api-access-q2vt7\") pod \"error-404-isvc-ae135-predictor-5dfb866474-tfrnf\" (UID: \"2838a1ec-8335-4af1-8956-fd80ed486104\") " pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:07.270732 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.270711    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2838a1ec-8335-4af1-8956-fd80ed486104-proxy-tls\") pod \"error-404-isvc-ae135-predictor-5dfb866474-tfrnf\" (UID: \"2838a1ec-8335-4af1-8956-fd80ed486104\") " pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:07.270852 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:52:07.270832    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-ae135-predictor-serving-cert: secret "error-404-isvc-ae135-predictor-serving-cert" not found
Jul 02 18:52:07.270919 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:52:07.270910    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2838a1ec-8335-4af1-8956-fd80ed486104-proxy-tls podName:2838a1ec-8335-4af1-8956-fd80ed486104 nodeName:}" failed. No retries permitted until 2026-07-02 18:52:07.770892802 +0000 UTC m=+1319.840520312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2838a1ec-8335-4af1-8956-fd80ed486104-proxy-tls") pod "error-404-isvc-ae135-predictor-5dfb866474-tfrnf" (UID: "2838a1ec-8335-4af1-8956-fd80ed486104") : secret "error-404-isvc-ae135-predictor-serving-cert" not found
Jul 02 18:52:07.271176 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.271159    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2838a1ec-8335-4af1-8956-fd80ed486104-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ae135-predictor-5dfb866474-tfrnf\" (UID: \"2838a1ec-8335-4af1-8956-fd80ed486104\") " pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:07.281261 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.281238    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2vt7\" (UniqueName: \"kubernetes.io/projected/2838a1ec-8335-4af1-8956-fd80ed486104-kube-api-access-q2vt7\") pod \"error-404-isvc-ae135-predictor-5dfb866474-tfrnf\" (UID: \"2838a1ec-8335-4af1-8956-fd80ed486104\") " pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:07.686685 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.686651    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:52:07.774031 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.774003    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2838a1ec-8335-4af1-8956-fd80ed486104-proxy-tls\") pod \"error-404-isvc-ae135-predictor-5dfb866474-tfrnf\" (UID: \"2838a1ec-8335-4af1-8956-fd80ed486104\") " pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:07.776060 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.776041    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2838a1ec-8335-4af1-8956-fd80ed486104-proxy-tls\") pod \"error-404-isvc-ae135-predictor-5dfb866474-tfrnf\" (UID: \"2838a1ec-8335-4af1-8956-fd80ed486104\") " pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:07.991242 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:07.991142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:08.135958 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:08.135921    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"]
Jul 02 18:52:08.208460 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:08.208430    2575 generic.go:356] "Generic (PLEG): container finished" podID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerID="228c53648c78133a17d7bdbd34cf0a48476ae1ea4f758c503c5bec20eeb118b9" exitCode=2
Jul 02 18:52:08.208610 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:08.208503    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" event={"ID":"ad7220bc-f3d6-40c9-830c-35024c9c3b76","Type":"ContainerDied","Data":"228c53648c78133a17d7bdbd34cf0a48476ae1ea4f758c503c5bec20eeb118b9"}
Jul 02 18:52:08.209959 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:08.209938    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" event={"ID":"2838a1ec-8335-4af1-8956-fd80ed486104","Type":"ContainerStarted","Data":"ee475fa2909a2f2aa5bdbb7c9d7fa05025f47de9c947cedb96be24554dc4b5b0"}
Jul 02 18:52:09.215773 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:09.215737    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" event={"ID":"2838a1ec-8335-4af1-8956-fd80ed486104","Type":"ContainerStarted","Data":"27e12981e1e8cb87b0e9704051c4158b690099661d125daf20641cef521ed552"}
Jul 02 18:52:09.216179 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:09.215802    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" event={"ID":"2838a1ec-8335-4af1-8956-fd80ed486104","Type":"ContainerStarted","Data":"8e101bd1f5cf4bb5da32b886718aa87cc7fadb3c213bcb2998d680014a3cc31a"}
Jul 02 18:52:09.216179 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:09.215832    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:09.216179 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:09.215875    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:09.217180 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:09.217157    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 02 18:52:09.252870 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:09.252815    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" podStartSLOduration=2.252798453 podStartE2EDuration="2.252798453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:52:09.250002126 +0000 UTC m=+1321.319629653" watchObservedRunningTime="2026-07-02 18:52:09.252798453 +0000 UTC m=+1321.322425978"
Jul 02 18:52:10.220235 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:10.220168    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 02 18:52:10.487417 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:10.487391    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:52:10.596043 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:10.596007    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh78c\" (UniqueName: \"kubernetes.io/projected/ad7220bc-f3d6-40c9-830c-35024c9c3b76-kube-api-access-gh78c\") pod \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\" (UID: \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\") "
Jul 02 18:52:10.596043 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:10.596047    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad7220bc-f3d6-40c9-830c-35024c9c3b76-kube-rbac-proxy-sar-config\") pod \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\" (UID: \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\") "
Jul 02 18:52:10.596305 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:10.596080    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad7220bc-f3d6-40c9-830c-35024c9c3b76-proxy-tls\") pod \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\" (UID: \"ad7220bc-f3d6-40c9-830c-35024c9c3b76\") "
Jul 02 18:52:10.596487 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:10.596463    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7220bc-f3d6-40c9-830c-35024c9c3b76-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad7220bc-f3d6-40c9-830c-35024c9c3b76" (UID: "ad7220bc-f3d6-40c9-830c-35024c9c3b76"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:52:10.598256 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:10.598218    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad7220bc-f3d6-40c9-830c-35024c9c3b76-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad7220bc-f3d6-40c9-830c-35024c9c3b76" (UID: "ad7220bc-f3d6-40c9-830c-35024c9c3b76"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:52:10.598256 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:10.598243    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7220bc-f3d6-40c9-830c-35024c9c3b76-kube-api-access-gh78c" (OuterVolumeSpecName: "kube-api-access-gh78c") pod "ad7220bc-f3d6-40c9-830c-35024c9c3b76" (UID: "ad7220bc-f3d6-40c9-830c-35024c9c3b76"). InnerVolumeSpecName "kube-api-access-gh78c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:52:10.697488 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:10.697450    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh78c\" (UniqueName: \"kubernetes.io/projected/ad7220bc-f3d6-40c9-830c-35024c9c3b76-kube-api-access-gh78c\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:52:10.697488 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:10.697483    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad7220bc-f3d6-40c9-830c-35024c9c3b76-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:52:10.697488 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:10.697493    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad7220bc-f3d6-40c9-830c-35024c9c3b76-proxy-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 18:52:11.225156 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:11.225125    2575 generic.go:356] "Generic (PLEG): container finished" podID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerID="1c2273ea57feadd7495cd98488f07c60cbae5f1af452f753f82b400c12a40cc7" exitCode=0
Jul 02 18:52:11.225588 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:11.225177    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"
Jul 02 18:52:11.225588 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:11.225177    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" event={"ID":"ad7220bc-f3d6-40c9-830c-35024c9c3b76","Type":"ContainerDied","Data":"1c2273ea57feadd7495cd98488f07c60cbae5f1af452f753f82b400c12a40cc7"}
Jul 02 18:52:11.225588 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:11.225324    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf" event={"ID":"ad7220bc-f3d6-40c9-830c-35024c9c3b76","Type":"ContainerDied","Data":"6147edcfccdbba3cf8911cd3cc6e89da9aed463ef3c997522526e39bbe7dcc5f"}
Jul 02 18:52:11.225588 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:11.225354    2575 scope.go:117] "RemoveContainer" containerID="228c53648c78133a17d7bdbd34cf0a48476ae1ea4f758c503c5bec20eeb118b9"
Jul 02 18:52:11.234485 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:11.234464    2575 scope.go:117] "RemoveContainer" containerID="1c2273ea57feadd7495cd98488f07c60cbae5f1af452f753f82b400c12a40cc7"
Jul 02 18:52:11.241882 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:11.241866    2575 scope.go:117] "RemoveContainer" containerID="228c53648c78133a17d7bdbd34cf0a48476ae1ea4f758c503c5bec20eeb118b9"
Jul 02 18:52:11.242153 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:52:11.242129    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228c53648c78133a17d7bdbd34cf0a48476ae1ea4f758c503c5bec20eeb118b9\": container with ID starting with 228c53648c78133a17d7bdbd34cf0a48476ae1ea4f758c503c5bec20eeb118b9 not found: ID does not exist" containerID="228c53648c78133a17d7bdbd34cf0a48476ae1ea4f758c503c5bec20eeb118b9"
Jul 02 18:52:11.242353 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:11.242160    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228c53648c78133a17d7bdbd34cf0a48476ae1ea4f758c503c5bec20eeb118b9"} err="failed to get container status \"228c53648c78133a17d7bdbd34cf0a48476ae1ea4f758c503c5bec20eeb118b9\": rpc error: code = NotFound desc = could not find container \"228c53648c78133a17d7bdbd34cf0a48476ae1ea4f758c503c5bec20eeb118b9\": container with ID starting with 228c53648c78133a17d7bdbd34cf0a48476ae1ea4f758c503c5bec20eeb118b9 not found: ID does not exist"
Jul 02 18:52:11.242353 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:11.242177    2575 scope.go:117] "RemoveContainer" containerID="1c2273ea57feadd7495cd98488f07c60cbae5f1af452f753f82b400c12a40cc7"
Jul 02 18:52:11.242452 ip-10-0-140-68 kubenswrapper[2575]: E0702 18:52:11.242426    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2273ea57feadd7495cd98488f07c60cbae5f1af452f753f82b400c12a40cc7\": container with ID starting with 1c2273ea57feadd7495cd98488f07c60cbae5f1af452f753f82b400c12a40cc7 not found: ID does not exist" containerID="1c2273ea57feadd7495cd98488f07c60cbae5f1af452f753f82b400c12a40cc7"
Jul 02 18:52:11.242487 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:11.242447    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2273ea57feadd7495cd98488f07c60cbae5f1af452f753f82b400c12a40cc7"} err="failed to get container status \"1c2273ea57feadd7495cd98488f07c60cbae5f1af452f753f82b400c12a40cc7\": rpc error: code = NotFound desc = could not find container \"1c2273ea57feadd7495cd98488f07c60cbae5f1af452f753f82b400c12a40cc7\": container with ID starting with 1c2273ea57feadd7495cd98488f07c60cbae5f1af452f753f82b400c12a40cc7 not found: ID does not exist"
Jul 02 18:52:11.246725 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:11.246706    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"]
Jul 02 18:52:11.249849 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:11.249830    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-94e8c-predictor-5d54994ccd-dlmkf"]
Jul 02 18:52:12.488307 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:12.488273    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" path="/var/lib/kubelet/pods/ad7220bc-f3d6-40c9-830c-35024c9c3b76/volumes"
Jul 02 18:52:15.057358 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:15.057332    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 18:52:15.225271 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:15.225243    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:52:15.225762 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:15.225724    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 02 18:52:25.226620 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:25.226581    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 02 18:52:27.248120 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:27.248078    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:52:27.344155 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:27.344113    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:52:35.226481 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:35.226381    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 02 18:52:45.226473 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:45.226430    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 02 18:52:55.227269 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:52:55.227236    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 18:53:05.818089 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:53:05.818047    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:53:07.242400 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:53:07.242363    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 18:55:08.443718 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:55:08.443689    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 18:55:08.446050 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:55:08.446028    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 18:55:08.451874 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:55:08.451856    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 18:55:08.453954 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:55:08.453938    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 18:57:36.060295 ip-10-0-140-68 kubenswrapper[2575]: I0702 18:57:36.060210    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:00:08.468682 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:08.468650    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 19:00:08.471479 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:08.471456    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 19:00:08.476478 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:08.476455    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 19:00:08.479170 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:08.479154    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 19:00:41.814228 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:41.814179    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:00:42.051292 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.051259    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"]
Jul 02 19:00:42.051579 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.051567    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerName="kube-rbac-proxy"
Jul 02 19:00:42.051623 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.051580    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerName="kube-rbac-proxy"
Jul 02 19:00:42.051623 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.051596    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerName="kserve-container"
Jul 02 19:00:42.051623 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.051602    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerName="kserve-container"
Jul 02 19:00:42.051721 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.051654    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerName="kube-rbac-proxy"
Jul 02 19:00:42.051721 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.051664    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad7220bc-f3d6-40c9-830c-35024c9c3b76" containerName="kserve-container"
Jul 02 19:00:42.054689 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.054674    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:00:42.057318 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.057297    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c532a-predictor-serving-cert\""
Jul 02 19:00:42.057550 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.057533    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c532a-kube-rbac-proxy-sar-config\""
Jul 02 19:00:42.064280 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.064219    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"]
Jul 02 19:00:42.083053 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.083028    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"]
Jul 02 19:00:42.083370 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.083314    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kserve-container" containerID="cri-o://708f23dad50eb619419dd18f1cf46c2428f2aea67b4e7a99a66ca3d691e780ae" gracePeriod=30
Jul 02 19:00:42.083499 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.083341    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kube-rbac-proxy" containerID="cri-o://ed3964c08b089fcc2259321cb5831632b56a4c3b8a76e90794b29a9aa7669476" gracePeriod=30
Jul 02 19:00:42.145785 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.145750    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2qv8\" (UniqueName: \"kubernetes.io/projected/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-kube-api-access-x2qv8\") pod \"error-404-isvc-c532a-predictor-78564c9984-cltvv\" (UID: \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\") " pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:00:42.145987 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.145812    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c532a-predictor-78564c9984-cltvv\" (UID: \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\") " pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:00:42.145987 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.145869    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-proxy-tls\") pod \"error-404-isvc-c532a-predictor-78564c9984-cltvv\" (UID: \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\") " pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:00:42.240143 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.240110    2575 generic.go:356] "Generic (PLEG): container finished" podID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerID="ed3964c08b089fcc2259321cb5831632b56a4c3b8a76e90794b29a9aa7669476" exitCode=2
Jul 02 19:00:42.240327 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.240203    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" event={"ID":"147ea518-52f2-41bf-8426-7f55a55d04d5","Type":"ContainerDied","Data":"ed3964c08b089fcc2259321cb5831632b56a4c3b8a76e90794b29a9aa7669476"}
Jul 02 19:00:42.246500 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.246481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c532a-predictor-78564c9984-cltvv\" (UID: \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\") " pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:00:42.246562 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.246512    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-proxy-tls\") pod \"error-404-isvc-c532a-predictor-78564c9984-cltvv\" (UID: \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\") " pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:00:42.246601 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.246581    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2qv8\" (UniqueName: \"kubernetes.io/projected/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-kube-api-access-x2qv8\") pod \"error-404-isvc-c532a-predictor-78564c9984-cltvv\" (UID: \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\") " pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:00:42.247129 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.247109    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c532a-predictor-78564c9984-cltvv\" (UID: \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\") " pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:00:42.248923 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.248905    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-proxy-tls\") pod \"error-404-isvc-c532a-predictor-78564c9984-cltvv\" (UID: \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\") " pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:00:42.254805 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.254785    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2qv8\" (UniqueName: \"kubernetes.io/projected/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-kube-api-access-x2qv8\") pod \"error-404-isvc-c532a-predictor-78564c9984-cltvv\" (UID: \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\") " pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:00:42.365726 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.365648    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:00:42.496267 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:42.496242    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"]
Jul 02 19:00:42.498128 ip-10-0-140-68 kubenswrapper[2575]: W0702 19:00:42.498096    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efd3f46_02ad_4da3_bbb3_1e889542a4c4.slice/crio-5e43d980e5e59693268083162e3186d9e2e3965d96f028f38298653434ddded1 WatchSource:0}: Error finding container 5e43d980e5e59693268083162e3186d9e2e3965d96f028f38298653434ddded1: Status 404 returned error can't find the container with id 5e43d980e5e59693268083162e3186d9e2e3965d96f028f38298653434ddded1
Jul 02 19:00:43.246326 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:43.246273    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" event={"ID":"4efd3f46-02ad-4da3-bbb3-1e889542a4c4","Type":"ContainerStarted","Data":"6d6db78cabd1f4c600309ecf0d2dd06ee3774255f7483bb872e4741a7d2bdc28"}
Jul 02 19:00:43.246768 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:43.246344    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" event={"ID":"4efd3f46-02ad-4da3-bbb3-1e889542a4c4","Type":"ContainerStarted","Data":"3136f8b76bddedcf915fc2ec75a354f44277792ac8c850d2e3987640f05d06cd"}
Jul 02 19:00:43.246768 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:43.246367    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" event={"ID":"4efd3f46-02ad-4da3-bbb3-1e889542a4c4","Type":"ContainerStarted","Data":"5e43d980e5e59693268083162e3186d9e2e3965d96f028f38298653434ddded1"}
Jul 02 19:00:43.246768 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:43.246391    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:00:43.246768 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:43.246420    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:00:43.247798 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:43.247770    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 02 19:00:43.266316 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:43.266270    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" podStartSLOduration=1.266256105 podStartE2EDuration="1.266256105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:00:43.263480587 +0000 UTC m=+1835.333108114" watchObservedRunningTime="2026-07-02 19:00:43.266256105 +0000 UTC m=+1835.335883633"
Jul 02 19:00:44.145889 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:44.145853    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:00:44.250775 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:44.250739    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 02 19:00:45.051643 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:45.051602    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 02 19:00:45.056430 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:45.056403    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 19:00:45.532238 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:45.532203    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 19:00:45.674062 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:45.673981    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjrc4\" (UniqueName: \"kubernetes.io/projected/147ea518-52f2-41bf-8426-7f55a55d04d5-kube-api-access-kjrc4\") pod \"147ea518-52f2-41bf-8426-7f55a55d04d5\" (UID: \"147ea518-52f2-41bf-8426-7f55a55d04d5\") "
Jul 02 19:00:45.674236 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:45.674088    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147ea518-52f2-41bf-8426-7f55a55d04d5-proxy-tls\") pod \"147ea518-52f2-41bf-8426-7f55a55d04d5\" (UID: \"147ea518-52f2-41bf-8426-7f55a55d04d5\") "
Jul 02 19:00:45.674236 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:45.674126    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147ea518-52f2-41bf-8426-7f55a55d04d5-kube-rbac-proxy-sar-config\") pod \"147ea518-52f2-41bf-8426-7f55a55d04d5\" (UID: \"147ea518-52f2-41bf-8426-7f55a55d04d5\") "
Jul 02 19:00:45.674502 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:45.674475    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147ea518-52f2-41bf-8426-7f55a55d04d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "147ea518-52f2-41bf-8426-7f55a55d04d5" (UID: "147ea518-52f2-41bf-8426-7f55a55d04d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 19:00:45.675944 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:45.675916    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147ea518-52f2-41bf-8426-7f55a55d04d5-kube-api-access-kjrc4" (OuterVolumeSpecName: "kube-api-access-kjrc4") pod "147ea518-52f2-41bf-8426-7f55a55d04d5" (UID: "147ea518-52f2-41bf-8426-7f55a55d04d5"). InnerVolumeSpecName "kube-api-access-kjrc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 19:00:45.676108 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:45.676091    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/147ea518-52f2-41bf-8426-7f55a55d04d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "147ea518-52f2-41bf-8426-7f55a55d04d5" (UID: "147ea518-52f2-41bf-8426-7f55a55d04d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 19:00:45.775163 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:45.775131    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147ea518-52f2-41bf-8426-7f55a55d04d5-proxy-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:00:45.775163 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:45.775159    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147ea518-52f2-41bf-8426-7f55a55d04d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:00:45.775163 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:45.775170    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjrc4\" (UniqueName: \"kubernetes.io/projected/147ea518-52f2-41bf-8426-7f55a55d04d5-kube-api-access-kjrc4\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:00:46.259787 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:46.259749    2575 generic.go:356] "Generic (PLEG): container finished" podID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerID="708f23dad50eb619419dd18f1cf46c2428f2aea67b4e7a99a66ca3d691e780ae" exitCode=0
Jul 02 19:00:46.259959 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:46.259806    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"
Jul 02 19:00:46.259959 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:46.259834    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" event={"ID":"147ea518-52f2-41bf-8426-7f55a55d04d5","Type":"ContainerDied","Data":"708f23dad50eb619419dd18f1cf46c2428f2aea67b4e7a99a66ca3d691e780ae"}
Jul 02 19:00:46.259959 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:46.259923    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l" event={"ID":"147ea518-52f2-41bf-8426-7f55a55d04d5","Type":"ContainerDied","Data":"f91cecd6e4e6ef8e242d51bd0c561f6ac8646bc5b01c6bac1d7060c99680e036"}
Jul 02 19:00:46.259959 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:46.259949    2575 scope.go:117] "RemoveContainer" containerID="ed3964c08b089fcc2259321cb5831632b56a4c3b8a76e90794b29a9aa7669476"
Jul 02 19:00:46.268340 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:46.268325    2575 scope.go:117] "RemoveContainer" containerID="708f23dad50eb619419dd18f1cf46c2428f2aea67b4e7a99a66ca3d691e780ae"
Jul 02 19:00:46.275415 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:46.275396    2575 scope.go:117] "RemoveContainer" containerID="ed3964c08b089fcc2259321cb5831632b56a4c3b8a76e90794b29a9aa7669476"
Jul 02 19:00:46.275641 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:00:46.275621    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3964c08b089fcc2259321cb5831632b56a4c3b8a76e90794b29a9aa7669476\": container with ID starting with ed3964c08b089fcc2259321cb5831632b56a4c3b8a76e90794b29a9aa7669476 not found: ID does not exist" containerID="ed3964c08b089fcc2259321cb5831632b56a4c3b8a76e90794b29a9aa7669476"
Jul 02 19:00:46.275686 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:46.275650    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3964c08b089fcc2259321cb5831632b56a4c3b8a76e90794b29a9aa7669476"} err="failed to get container status \"ed3964c08b089fcc2259321cb5831632b56a4c3b8a76e90794b29a9aa7669476\": rpc error: code = NotFound desc = could not find container \"ed3964c08b089fcc2259321cb5831632b56a4c3b8a76e90794b29a9aa7669476\": container with ID starting with ed3964c08b089fcc2259321cb5831632b56a4c3b8a76e90794b29a9aa7669476 not found: ID does not exist"
Jul 02 19:00:46.275686 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:46.275666    2575 scope.go:117] "RemoveContainer" containerID="708f23dad50eb619419dd18f1cf46c2428f2aea67b4e7a99a66ca3d691e780ae"
Jul 02 19:00:46.275885 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:00:46.275868    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708f23dad50eb619419dd18f1cf46c2428f2aea67b4e7a99a66ca3d691e780ae\": container with ID starting with 708f23dad50eb619419dd18f1cf46c2428f2aea67b4e7a99a66ca3d691e780ae not found: ID does not exist" containerID="708f23dad50eb619419dd18f1cf46c2428f2aea67b4e7a99a66ca3d691e780ae"
Jul 02 19:00:46.275934 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:46.275894    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708f23dad50eb619419dd18f1cf46c2428f2aea67b4e7a99a66ca3d691e780ae"} err="failed to get container status \"708f23dad50eb619419dd18f1cf46c2428f2aea67b4e7a99a66ca3d691e780ae\": rpc error: code = NotFound desc = could not find container \"708f23dad50eb619419dd18f1cf46c2428f2aea67b4e7a99a66ca3d691e780ae\": container with ID starting with 708f23dad50eb619419dd18f1cf46c2428f2aea67b4e7a99a66ca3d691e780ae not found: ID does not exist"
Jul 02 19:00:46.280939 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:46.280917    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"]
Jul 02 19:00:46.286441 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:46.286421    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7acc8-predictor-7cf8fdd8fb-hk57l"]
Jul 02 19:00:46.488516 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:46.488486    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" path="/var/lib/kubelet/pods/147ea518-52f2-41bf-8426-7f55a55d04d5/volumes"
Jul 02 19:00:49.255696 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:49.255667    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:00:49.256114 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:49.256088    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 02 19:00:59.256810 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:00:59.256771    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 02 19:01:09.256737 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:09.256700    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 02 19:01:19.256790 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:19.256749    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 02 19:01:21.850787 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:21.850751    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:01:21.907437 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:21.907398    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"]
Jul 02 19:01:21.907759 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:21.907697    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kserve-container" containerID="cri-o://27e12981e1e8cb87b0e9704051c4158b690099661d125daf20641cef521ed552" gracePeriod=30
Jul 02 19:01:21.907876 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:21.907771    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kube-rbac-proxy" containerID="cri-o://8e101bd1f5cf4bb5da32b886718aa87cc7fadb3c213bcb2998d680014a3cc31a" gracePeriod=30
Jul 02 19:01:21.937656 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:21.937625    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:01:22.004159 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.004129    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:01:22.079570 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.079536    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:01:22.080454 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.080435    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"]
Jul 02 19:01:22.080744 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.080732    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kube-rbac-proxy"
Jul 02 19:01:22.080782 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.080745    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kube-rbac-proxy"
Jul 02 19:01:22.080782 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.080765    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kserve-container"
Jul 02 19:01:22.080782 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.080770    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kserve-container"
Jul 02 19:01:22.080872 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.080820    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kube-rbac-proxy"
Jul 02 19:01:22.080872 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.080831    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="147ea518-52f2-41bf-8426-7f55a55d04d5" containerName="kserve-container"
Jul 02 19:01:22.083789 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.083772    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:01:22.086635 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.086617    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-208dc-kube-rbac-proxy-sar-config\""
Jul 02 19:01:22.086725 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.086646    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-208dc-predictor-serving-cert\""
Jul 02 19:01:22.091479 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.091455    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"]
Jul 02 19:01:22.178877 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.178779    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f89b4832-6ab6-47ea-b907-00ea9a6d239b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4\" (UID: \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\") " pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:01:22.178877 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.178838    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f89b4832-6ab6-47ea-b907-00ea9a6d239b-proxy-tls\") pod \"error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4\" (UID: \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\") " pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:01:22.179080 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.178909    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj7bf\" (UniqueName: \"kubernetes.io/projected/f89b4832-6ab6-47ea-b907-00ea9a6d239b-kube-api-access-bj7bf\") pod \"error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4\" (UID: \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\") " pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:01:22.190536 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.190505    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:01:22.279443 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.279409    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj7bf\" (UniqueName: \"kubernetes.io/projected/f89b4832-6ab6-47ea-b907-00ea9a6d239b-kube-api-access-bj7bf\") pod \"error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4\" (UID: \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\") " pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:01:22.279618 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.279467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f89b4832-6ab6-47ea-b907-00ea9a6d239b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4\" (UID: \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\") " pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:01:22.279618 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.279511    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f89b4832-6ab6-47ea-b907-00ea9a6d239b-proxy-tls\") pod \"error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4\" (UID: \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\") " pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:01:22.280123 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.280094    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f89b4832-6ab6-47ea-b907-00ea9a6d239b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4\" (UID: \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\") " pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:01:22.281927 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.281904    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f89b4832-6ab6-47ea-b907-00ea9a6d239b-proxy-tls\") pod \"error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4\" (UID: \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\") " pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:01:22.287315 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.287294    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj7bf\" (UniqueName: \"kubernetes.io/projected/f89b4832-6ab6-47ea-b907-00ea9a6d239b-kube-api-access-bj7bf\") pod \"error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4\" (UID: \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\") " pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:01:22.394298 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.394266    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:01:22.404837 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.404727    2575 generic.go:356] "Generic (PLEG): container finished" podID="2838a1ec-8335-4af1-8956-fd80ed486104" containerID="8e101bd1f5cf4bb5da32b886718aa87cc7fadb3c213bcb2998d680014a3cc31a" exitCode=2
Jul 02 19:01:22.404837 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.404800    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" event={"ID":"2838a1ec-8335-4af1-8956-fd80ed486104","Type":"ContainerDied","Data":"8e101bd1f5cf4bb5da32b886718aa87cc7fadb3c213bcb2998d680014a3cc31a"}
Jul 02 19:01:22.430563 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.430465    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:01:22.523485 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:22.523454    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"]
Jul 02 19:01:22.525589 ip-10-0-140-68 kubenswrapper[2575]: W0702 19:01:22.525562    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89b4832_6ab6_47ea_b907_00ea9a6d239b.slice/crio-431f378f82d776329f571e6899b01296b5a3daa44b3d52f0589ab75858802445 WatchSource:0}: Error finding container 431f378f82d776329f571e6899b01296b5a3daa44b3d52f0589ab75858802445: Status 404 returned error can't find the container with id 431f378f82d776329f571e6899b01296b5a3daa44b3d52f0589ab75858802445
Jul 02 19:01:23.410278 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:23.410239    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" event={"ID":"f89b4832-6ab6-47ea-b907-00ea9a6d239b","Type":"ContainerStarted","Data":"cd4e8089943da08010e0db8048a893ae1e4eb61cd3118c0983c5bd47376d4e02"}
Jul 02 19:01:23.410744 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:23.410295    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" event={"ID":"f89b4832-6ab6-47ea-b907-00ea9a6d239b","Type":"ContainerStarted","Data":"9693586d46f8451a73bf503aabc9584373824cdfaa7a6f55df24617ca648cb3f"}
Jul 02 19:01:23.410744 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:23.410317    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" event={"ID":"f89b4832-6ab6-47ea-b907-00ea9a6d239b","Type":"ContainerStarted","Data":"431f378f82d776329f571e6899b01296b5a3daa44b3d52f0589ab75858802445"}
Jul 02 19:01:23.410744 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:23.410356    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:01:23.428655 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:23.428539    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" podStartSLOduration=1.428517263 podStartE2EDuration="1.428517263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:01:23.427255934 +0000 UTC m=+1875.496883464" watchObservedRunningTime="2026-07-02 19:01:23.428517263 +0000 UTC m=+1875.498144794"
Jul 02 19:01:24.414511 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:24.414475    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:01:24.415954 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:24.415925    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 19:01:25.220685 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.220645    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 02 19:01:25.226121 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.226091    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 02 19:01:25.359489 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.359465    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 19:01:25.419469 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.419382    2575 generic.go:356] "Generic (PLEG): container finished" podID="2838a1ec-8335-4af1-8956-fd80ed486104" containerID="27e12981e1e8cb87b0e9704051c4158b690099661d125daf20641cef521ed552" exitCode=0
Jul 02 19:01:25.419469 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.419431    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"
Jul 02 19:01:25.419947 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.419461    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" event={"ID":"2838a1ec-8335-4af1-8956-fd80ed486104","Type":"ContainerDied","Data":"27e12981e1e8cb87b0e9704051c4158b690099661d125daf20641cef521ed552"}
Jul 02 19:01:25.419947 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.419522    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf" event={"ID":"2838a1ec-8335-4af1-8956-fd80ed486104","Type":"ContainerDied","Data":"ee475fa2909a2f2aa5bdbb7c9d7fa05025f47de9c947cedb96be24554dc4b5b0"}
Jul 02 19:01:25.419947 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.419549    2575 scope.go:117] "RemoveContainer" containerID="8e101bd1f5cf4bb5da32b886718aa87cc7fadb3c213bcb2998d680014a3cc31a"
Jul 02 19:01:25.420105 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.419946    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 19:01:25.427658 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.427588    2575 scope.go:117] "RemoveContainer" containerID="27e12981e1e8cb87b0e9704051c4158b690099661d125daf20641cef521ed552"
Jul 02 19:01:25.435650 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.435624    2575 scope.go:117] "RemoveContainer" containerID="8e101bd1f5cf4bb5da32b886718aa87cc7fadb3c213bcb2998d680014a3cc31a"
Jul 02 19:01:25.435973 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:01:25.435954    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e101bd1f5cf4bb5da32b886718aa87cc7fadb3c213bcb2998d680014a3cc31a\": container with ID starting with 8e101bd1f5cf4bb5da32b886718aa87cc7fadb3c213bcb2998d680014a3cc31a not found: ID does not exist" containerID="8e101bd1f5cf4bb5da32b886718aa87cc7fadb3c213bcb2998d680014a3cc31a"
Jul 02 19:01:25.436042 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.435981    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e101bd1f5cf4bb5da32b886718aa87cc7fadb3c213bcb2998d680014a3cc31a"} err="failed to get container status \"8e101bd1f5cf4bb5da32b886718aa87cc7fadb3c213bcb2998d680014a3cc31a\": rpc error: code = NotFound desc = could not find container \"8e101bd1f5cf4bb5da32b886718aa87cc7fadb3c213bcb2998d680014a3cc31a\": container with ID starting with 8e101bd1f5cf4bb5da32b886718aa87cc7fadb3c213bcb2998d680014a3cc31a not found: ID does not exist"
Jul 02 19:01:25.436042 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.435999    2575 scope.go:117] "RemoveContainer" containerID="27e12981e1e8cb87b0e9704051c4158b690099661d125daf20641cef521ed552"
Jul 02 19:01:25.436326 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:01:25.436301    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e12981e1e8cb87b0e9704051c4158b690099661d125daf20641cef521ed552\": container with ID starting with 27e12981e1e8cb87b0e9704051c4158b690099661d125daf20641cef521ed552 not found: ID does not exist" containerID="27e12981e1e8cb87b0e9704051c4158b690099661d125daf20641cef521ed552"
Jul 02 19:01:25.436426 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.436335    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e12981e1e8cb87b0e9704051c4158b690099661d125daf20641cef521ed552"} err="failed to get container status \"27e12981e1e8cb87b0e9704051c4158b690099661d125daf20641cef521ed552\": rpc error: code = NotFound desc = could not find container \"27e12981e1e8cb87b0e9704051c4158b690099661d125daf20641cef521ed552\": container with ID starting with 27e12981e1e8cb87b0e9704051c4158b690099661d125daf20641cef521ed552 not found: ID does not exist"
Jul 02 19:01:25.508207 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.508160    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2838a1ec-8335-4af1-8956-fd80ed486104-proxy-tls\") pod \"2838a1ec-8335-4af1-8956-fd80ed486104\" (UID: \"2838a1ec-8335-4af1-8956-fd80ed486104\") "
Jul 02 19:01:25.508419 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.508299    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2838a1ec-8335-4af1-8956-fd80ed486104-kube-rbac-proxy-sar-config\") pod \"2838a1ec-8335-4af1-8956-fd80ed486104\" (UID: \"2838a1ec-8335-4af1-8956-fd80ed486104\") "
Jul 02 19:01:25.508419 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.508355    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2vt7\" (UniqueName: \"kubernetes.io/projected/2838a1ec-8335-4af1-8956-fd80ed486104-kube-api-access-q2vt7\") pod \"2838a1ec-8335-4af1-8956-fd80ed486104\" (UID: \"2838a1ec-8335-4af1-8956-fd80ed486104\") "
Jul 02 19:01:25.508639 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.508615    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2838a1ec-8335-4af1-8956-fd80ed486104-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2838a1ec-8335-4af1-8956-fd80ed486104" (UID: "2838a1ec-8335-4af1-8956-fd80ed486104"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 19:01:25.510205 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.510163    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2838a1ec-8335-4af1-8956-fd80ed486104-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2838a1ec-8335-4af1-8956-fd80ed486104" (UID: "2838a1ec-8335-4af1-8956-fd80ed486104"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 19:01:25.510351 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.510321    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2838a1ec-8335-4af1-8956-fd80ed486104-kube-api-access-q2vt7" (OuterVolumeSpecName: "kube-api-access-q2vt7") pod "2838a1ec-8335-4af1-8956-fd80ed486104" (UID: "2838a1ec-8335-4af1-8956-fd80ed486104"). InnerVolumeSpecName "kube-api-access-q2vt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 19:01:25.609039 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.609002    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2838a1ec-8335-4af1-8956-fd80ed486104-proxy-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:01:25.609039 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.609032    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2838a1ec-8335-4af1-8956-fd80ed486104-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:01:25.609039 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.609041    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2vt7\" (UniqueName: \"kubernetes.io/projected/2838a1ec-8335-4af1-8956-fd80ed486104-kube-api-access-q2vt7\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:01:25.743069 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.743032    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"]
Jul 02 19:01:25.744659 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:25.744635    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ae135-predictor-5dfb866474-tfrnf"]
Jul 02 19:01:26.493942 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:26.493910    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" path="/var/lib/kubelet/pods/2838a1ec-8335-4af1-8956-fd80ed486104/volumes"
Jul 02 19:01:29.256341 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:29.256311    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:01:30.424727 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:30.424643    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:01:30.425184 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:30.425065    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 19:01:40.425099 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:40.425051    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 19:01:41.983308 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:41.983273    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:01:42.159880 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:42.159849    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:01:50.425976 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:50.425930    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 19:01:52.077337 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.077306    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:01:52.207764 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.207727    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:01:52.277905 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.277873    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"]
Jul 02 19:01:52.278159 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.278133    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kserve-container" containerID="cri-o://3136f8b76bddedcf915fc2ec75a354f44277792ac8c850d2e3987640f05d06cd" gracePeriod=30
Jul 02 19:01:52.278290 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.278243    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kube-rbac-proxy" containerID="cri-o://6d6db78cabd1f4c600309ecf0d2dd06ee3774255f7483bb872e4741a7d2bdc28" gracePeriod=30
Jul 02 19:01:52.282846 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.282825    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:01:52.296361 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.296336    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"]
Jul 02 19:01:52.296647 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.296635    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kube-rbac-proxy"
Jul 02 19:01:52.296689 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.296649    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kube-rbac-proxy"
Jul 02 19:01:52.296689 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.296658    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kserve-container"
Jul 02 19:01:52.296689 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.296664    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kserve-container"
Jul 02 19:01:52.296785 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.296723    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kserve-container"
Jul 02 19:01:52.296785 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.296734    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2838a1ec-8335-4af1-8956-fd80ed486104" containerName="kube-rbac-proxy"
Jul 02 19:01:52.299962 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.299943    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:01:52.302940 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.302921    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c41f9-predictor-serving-cert\""
Jul 02 19:01:52.303048 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.303029    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c41f9-kube-rbac-proxy-sar-config\""
Jul 02 19:01:52.309312 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.309288    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"]
Jul 02 19:01:52.326448 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.326332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-proxy-tls\") pod \"error-404-isvc-c41f9-predictor-78b9f9dc78-glpck\" (UID: \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\") " pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:01:52.326448 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.326434    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5fqn\" (UniqueName: \"kubernetes.io/projected/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-kube-api-access-v5fqn\") pod \"error-404-isvc-c41f9-predictor-78b9f9dc78-glpck\" (UID: \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\") " pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:01:52.326676 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.326474    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c41f9-predictor-78b9f9dc78-glpck\" (UID: \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\") " pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:01:52.382752 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.382676    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:01:52.427121 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.427071    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5fqn\" (UniqueName: \"kubernetes.io/projected/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-kube-api-access-v5fqn\") pod \"error-404-isvc-c41f9-predictor-78b9f9dc78-glpck\" (UID: \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\") " pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:01:52.427367 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.427129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c41f9-predictor-78b9f9dc78-glpck\" (UID: \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\") " pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:01:52.427367 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.427215    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-proxy-tls\") pod \"error-404-isvc-c41f9-predictor-78b9f9dc78-glpck\" (UID: \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\") " pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:01:52.427481 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:01:52.427371    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-serving-cert: secret "error-404-isvc-c41f9-predictor-serving-cert" not found
Jul 02 19:01:52.427481 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:01:52.427446    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-proxy-tls podName:3a1f430b-1817-4fd5-aeef-7d0b4b354bef nodeName:}" failed. No retries permitted until 2026-07-02 19:01:52.927424359 +0000 UTC m=+1904.997051892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-proxy-tls") pod "error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" (UID: "3a1f430b-1817-4fd5-aeef-7d0b4b354bef") : secret "error-404-isvc-c41f9-predictor-serving-cert" not found
Jul 02 19:01:52.428000 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.427977    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c41f9-predictor-78b9f9dc78-glpck\" (UID: \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\") " pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:01:52.438044 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.438017    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5fqn\" (UniqueName: \"kubernetes.io/projected/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-kube-api-access-v5fqn\") pod \"error-404-isvc-c41f9-predictor-78b9f9dc78-glpck\" (UID: \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\") " pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:01:52.533373 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.533329    2575 generic.go:356] "Generic (PLEG): container finished" podID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerID="6d6db78cabd1f4c600309ecf0d2dd06ee3774255f7483bb872e4741a7d2bdc28" exitCode=2
Jul 02 19:01:52.533529 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.533397    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" event={"ID":"4efd3f46-02ad-4da3-bbb3-1e889542a4c4","Type":"ContainerDied","Data":"6d6db78cabd1f4c600309ecf0d2dd06ee3774255f7483bb872e4741a7d2bdc28"}
Jul 02 19:01:52.654517 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.654416    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:01:52.931941 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.931831    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-proxy-tls\") pod \"error-404-isvc-c41f9-predictor-78b9f9dc78-glpck\" (UID: \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\") " pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:01:52.934286 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:52.934259    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-proxy-tls\") pod \"error-404-isvc-c41f9-predictor-78b9f9dc78-glpck\" (UID: \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\") " pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:01:53.209862 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:53.209771    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:01:53.336377 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:53.336205    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"]
Jul 02 19:01:53.339010 ip-10-0-140-68 kubenswrapper[2575]: W0702 19:01:53.338979    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1f430b_1817_4fd5_aeef_7d0b4b354bef.slice/crio-883ce1f1b86d11d6e6b43ad984f53134734b6f998802e7a36e928261bda8ee23 WatchSource:0}: Error finding container 883ce1f1b86d11d6e6b43ad984f53134734b6f998802e7a36e928261bda8ee23: Status 404 returned error can't find the container with id 883ce1f1b86d11d6e6b43ad984f53134734b6f998802e7a36e928261bda8ee23
Jul 02 19:01:53.540551 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:53.540465    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" event={"ID":"3a1f430b-1817-4fd5-aeef-7d0b4b354bef","Type":"ContainerStarted","Data":"3342cee40718ebd95e589ab80369b0982d7234f396424358680c05ce5782446e"}
Jul 02 19:01:53.540551 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:53.540522    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" event={"ID":"3a1f430b-1817-4fd5-aeef-7d0b4b354bef","Type":"ContainerStarted","Data":"75cfe522c5267186ea6191732dc14045a8ebf6f944bea669c0374483bcdb9395"}
Jul 02 19:01:53.540551 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:53.540543    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" event={"ID":"3a1f430b-1817-4fd5-aeef-7d0b4b354bef","Type":"ContainerStarted","Data":"883ce1f1b86d11d6e6b43ad984f53134734b6f998802e7a36e928261bda8ee23"}
Jul 02 19:01:53.540772 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:53.540570    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:01:53.559583 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:53.559541    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" podStartSLOduration=1.559528141 podStartE2EDuration="1.559528141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:01:53.557951773 +0000 UTC m=+1905.627579297" watchObservedRunningTime="2026-07-02 19:01:53.559528141 +0000 UTC m=+1905.629155669"
Jul 02 19:01:54.250946 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:54.250903    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 02 19:01:54.544773 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:54.544677    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:01:54.545922 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:54.545886    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 19:01:55.551206 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:55.551159    2575 generic.go:356] "Generic (PLEG): container finished" podID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerID="3136f8b76bddedcf915fc2ec75a354f44277792ac8c850d2e3987640f05d06cd" exitCode=0
Jul 02 19:01:55.551611 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:55.551226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" event={"ID":"4efd3f46-02ad-4da3-bbb3-1e889542a4c4","Type":"ContainerDied","Data":"3136f8b76bddedcf915fc2ec75a354f44277792ac8c850d2e3987640f05d06cd"}
Jul 02 19:01:55.551791 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:55.551761    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 19:01:55.625384 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:55.625358    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:01:55.652238 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:55.652171    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-proxy-tls\") pod \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\" (UID: \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\") "
Jul 02 19:01:55.652388 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:55.652286    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2qv8\" (UniqueName: \"kubernetes.io/projected/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-kube-api-access-x2qv8\") pod \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\" (UID: \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\") "
Jul 02 19:01:55.652430 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:55.652397    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-kube-rbac-proxy-sar-config\") pod \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\" (UID: \"4efd3f46-02ad-4da3-bbb3-1e889542a4c4\") "
Jul 02 19:01:55.652832 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:55.652787    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4efd3f46-02ad-4da3-bbb3-1e889542a4c4" (UID: "4efd3f46-02ad-4da3-bbb3-1e889542a4c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 19:01:55.654302 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:55.654268    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4efd3f46-02ad-4da3-bbb3-1e889542a4c4" (UID: "4efd3f46-02ad-4da3-bbb3-1e889542a4c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 19:01:55.654400 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:55.654301    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-kube-api-access-x2qv8" (OuterVolumeSpecName: "kube-api-access-x2qv8") pod "4efd3f46-02ad-4da3-bbb3-1e889542a4c4" (UID: "4efd3f46-02ad-4da3-bbb3-1e889542a4c4"). InnerVolumeSpecName "kube-api-access-x2qv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 19:01:55.753370 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:55.753294    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2qv8\" (UniqueName: \"kubernetes.io/projected/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-kube-api-access-x2qv8\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:01:55.753370 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:55.753319    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:01:55.753370 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:55.753328    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4efd3f46-02ad-4da3-bbb3-1e889542a4c4-proxy-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:01:56.558923 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:56.558896    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"
Jul 02 19:01:56.558923 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:56.558890    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv" event={"ID":"4efd3f46-02ad-4da3-bbb3-1e889542a4c4","Type":"ContainerDied","Data":"5e43d980e5e59693268083162e3186d9e2e3965d96f028f38298653434ddded1"}
Jul 02 19:01:56.559489 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:56.558980    2575 scope.go:117] "RemoveContainer" containerID="6d6db78cabd1f4c600309ecf0d2dd06ee3774255f7483bb872e4741a7d2bdc28"
Jul 02 19:01:56.566973 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:56.566958    2575 scope.go:117] "RemoveContainer" containerID="3136f8b76bddedcf915fc2ec75a354f44277792ac8c850d2e3987640f05d06cd"
Jul 02 19:01:56.575767 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:56.575727    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"]
Jul 02 19:01:56.579078 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:56.579055    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c532a-predictor-78564c9984-cltvv"]
Jul 02 19:01:58.487597 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:01:58.487565    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" path="/var/lib/kubelet/pods/4efd3f46-02ad-4da3-bbb3-1e889542a4c4/volumes"
Jul 02 19:02:00.425390 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:02:00.425340    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 19:02:00.556300 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:02:00.556274    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:02:00.556663 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:02:00.556637    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 19:02:10.426347 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:02:10.426310    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:02:10.557321 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:02:10.557284    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 19:02:20.163809 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:02:20.163767    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:02:20.557332 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:02:20.557294    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 19:02:22.043108 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:02:22.043075    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:02:22.471009 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:02:22.470972    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:02:30.556819 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:02:30.556781    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 19:02:40.557661 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:02:40.557632    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:02:52.285084 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:02:52.285049    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:02:52.494319 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:02:52.494288    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:05:08.495021 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:05:08.494991    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 19:05:08.497833 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:05:08.497812    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 19:05:08.503373 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:05:08.503343    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 19:05:08.506293 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:05:08.506276    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 19:06:06.635222 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:06:06.635132    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:06:13.255344 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:06:13.255308    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:10:08.521324 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:10:08.521295    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 19:10:08.524453 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:10:08.524428    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 19:10:08.529187 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:10:08.529165    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 19:10:08.532202 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:10:08.532171    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 19:11:07.006953 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:07.006890    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:11:07.100972 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:07.100934    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:11:07.105820 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:07.105794    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"]
Jul 02 19:11:07.106111 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:07.106067    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerName="kserve-container" containerID="cri-o://75cfe522c5267186ea6191732dc14045a8ebf6f944bea669c0374483bcdb9395" gracePeriod=30
Jul 02 19:11:07.106342 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:07.106142    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerName="kube-rbac-proxy" containerID="cri-o://3342cee40718ebd95e589ab80369b0982d7234f396424358680c05ce5782446e" gracePeriod=30
Jul 02 19:11:07.152762 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:07.152715    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:11:07.697722 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:07.697682    2575 generic.go:356] "Generic (PLEG): container finished" podID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerID="3342cee40718ebd95e589ab80369b0982d7234f396424358680c05ce5782446e" exitCode=2
Jul 02 19:11:07.697900 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:07.697748    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" event={"ID":"3a1f430b-1817-4fd5-aeef-7d0b4b354bef","Type":"ContainerDied","Data":"3342cee40718ebd95e589ab80369b0982d7234f396424358680c05ce5782446e"}
Jul 02 19:11:10.453572 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.453547    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:11:10.472066 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.472042    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-proxy-tls\") pod \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\" (UID: \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\") "
Jul 02 19:11:10.472222 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.472106    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5fqn\" (UniqueName: \"kubernetes.io/projected/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-kube-api-access-v5fqn\") pod \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\" (UID: \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\") "
Jul 02 19:11:10.472222 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.472149    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-kube-rbac-proxy-sar-config\") pod \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\" (UID: \"3a1f430b-1817-4fd5-aeef-7d0b4b354bef\") "
Jul 02 19:11:10.472565 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.472539    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a1f430b-1817-4fd5-aeef-7d0b4b354bef" (UID: "3a1f430b-1817-4fd5-aeef-7d0b4b354bef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 19:11:10.474242 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.474215    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a1f430b-1817-4fd5-aeef-7d0b4b354bef" (UID: "3a1f430b-1817-4fd5-aeef-7d0b4b354bef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 19:11:10.474544 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.474521    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-kube-api-access-v5fqn" (OuterVolumeSpecName: "kube-api-access-v5fqn") pod "3a1f430b-1817-4fd5-aeef-7d0b4b354bef" (UID: "3a1f430b-1817-4fd5-aeef-7d0b4b354bef"). InnerVolumeSpecName "kube-api-access-v5fqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 19:11:10.572893 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.572853    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5fqn\" (UniqueName: \"kubernetes.io/projected/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-kube-api-access-v5fqn\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:11:10.572893 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.572887    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:11:10.572893 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.572897    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a1f430b-1817-4fd5-aeef-7d0b4b354bef-proxy-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:11:10.711640 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.711604    2575 generic.go:356] "Generic (PLEG): container finished" podID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerID="75cfe522c5267186ea6191732dc14045a8ebf6f944bea669c0374483bcdb9395" exitCode=0
Jul 02 19:11:10.711827 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.711659    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"
Jul 02 19:11:10.711827 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.711686    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" event={"ID":"3a1f430b-1817-4fd5-aeef-7d0b4b354bef","Type":"ContainerDied","Data":"75cfe522c5267186ea6191732dc14045a8ebf6f944bea669c0374483bcdb9395"}
Jul 02 19:11:10.711827 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.711756    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck" event={"ID":"3a1f430b-1817-4fd5-aeef-7d0b4b354bef","Type":"ContainerDied","Data":"883ce1f1b86d11d6e6b43ad984f53134734b6f998802e7a36e928261bda8ee23"}
Jul 02 19:11:10.711827 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.711793    2575 scope.go:117] "RemoveContainer" containerID="3342cee40718ebd95e589ab80369b0982d7234f396424358680c05ce5782446e"
Jul 02 19:11:10.719784 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.719757    2575 scope.go:117] "RemoveContainer" containerID="75cfe522c5267186ea6191732dc14045a8ebf6f944bea669c0374483bcdb9395"
Jul 02 19:11:10.729474 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.728249    2575 scope.go:117] "RemoveContainer" containerID="3342cee40718ebd95e589ab80369b0982d7234f396424358680c05ce5782446e"
Jul 02 19:11:10.729648 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:11:10.729621    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3342cee40718ebd95e589ab80369b0982d7234f396424358680c05ce5782446e\": container with ID starting with 3342cee40718ebd95e589ab80369b0982d7234f396424358680c05ce5782446e not found: ID does not exist" containerID="3342cee40718ebd95e589ab80369b0982d7234f396424358680c05ce5782446e"
Jul 02 19:11:10.729739 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.729660    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3342cee40718ebd95e589ab80369b0982d7234f396424358680c05ce5782446e"} err="failed to get container status \"3342cee40718ebd95e589ab80369b0982d7234f396424358680c05ce5782446e\": rpc error: code = NotFound desc = could not find container \"3342cee40718ebd95e589ab80369b0982d7234f396424358680c05ce5782446e\": container with ID starting with 3342cee40718ebd95e589ab80369b0982d7234f396424358680c05ce5782446e not found: ID does not exist"
Jul 02 19:11:10.729739 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.729687    2575 scope.go:117] "RemoveContainer" containerID="75cfe522c5267186ea6191732dc14045a8ebf6f944bea669c0374483bcdb9395"
Jul 02 19:11:10.730145 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:11:10.730126    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75cfe522c5267186ea6191732dc14045a8ebf6f944bea669c0374483bcdb9395\": container with ID starting with 75cfe522c5267186ea6191732dc14045a8ebf6f944bea669c0374483bcdb9395 not found: ID does not exist" containerID="75cfe522c5267186ea6191732dc14045a8ebf6f944bea669c0374483bcdb9395"
Jul 02 19:11:10.730244 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.730149    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75cfe522c5267186ea6191732dc14045a8ebf6f944bea669c0374483bcdb9395"} err="failed to get container status \"75cfe522c5267186ea6191732dc14045a8ebf6f944bea669c0374483bcdb9395\": rpc error: code = NotFound desc = could not find container \"75cfe522c5267186ea6191732dc14045a8ebf6f944bea669c0374483bcdb9395\": container with ID starting with 75cfe522c5267186ea6191732dc14045a8ebf6f944bea669c0374483bcdb9395 not found: ID does not exist"
Jul 02 19:11:10.730567 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.730550    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"]
Jul 02 19:11:10.733388 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:10.733367    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c41f9-predictor-78b9f9dc78-glpck"]
Jul 02 19:11:12.489077 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:12.489037    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" path="/var/lib/kubelet/pods/3a1f430b-1817-4fd5-aeef-7d0b4b354bef/volumes"
Jul 02 19:11:51.554301 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:11:51.554264    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:12:46.466228 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:12:46.466177    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:15:08.545254 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:15:08.545121    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 19:15:08.549940 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:15:08.549922    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 19:15:08.553000 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:15:08.552984    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 19:15:08.557611 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:15:08.557595    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 19:18:11.033341 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:11.033255    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:18:41.461920 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:41.461879    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:18:41.597654 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:41.597618    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:18:41.632897 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:41.632862    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"]
Jul 02 19:18:41.633103 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:41.633077    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerName="kserve-container" containerID="cri-o://9693586d46f8451a73bf503aabc9584373824cdfaa7a6f55df24617ca648cb3f" gracePeriod=30
Jul 02 19:18:41.633176 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:41.633122    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerName="kube-rbac-proxy" containerID="cri-o://cd4e8089943da08010e0db8048a893ae1e4eb61cd3118c0983c5bd47376d4e02" gracePeriod=30
Jul 02 19:18:41.657448 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:41.657422    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:18:42.440407 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.440367    2575 generic.go:356] "Generic (PLEG): container finished" podID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerID="cd4e8089943da08010e0db8048a893ae1e4eb61cd3118c0983c5bd47376d4e02" exitCode=2
Jul 02 19:18:42.440664 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.440432    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" event={"ID":"f89b4832-6ab6-47ea-b907-00ea9a6d239b","Type":"ContainerDied","Data":"cd4e8089943da08010e0db8048a893ae1e4eb61cd3118c0983c5bd47376d4e02"}
Jul 02 19:18:42.871668 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.871636    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hw6zs/must-gather-d5f6t"]
Jul 02 19:18:42.872045 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.871920    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kube-rbac-proxy"
Jul 02 19:18:42.872045 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.871931    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kube-rbac-proxy"
Jul 02 19:18:42.872045 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.871941    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerName="kube-rbac-proxy"
Jul 02 19:18:42.872045 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.871947    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerName="kube-rbac-proxy"
Jul 02 19:18:42.872045 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.871962    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kserve-container"
Jul 02 19:18:42.872045 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.871968    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kserve-container"
Jul 02 19:18:42.872045 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.871980    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerName="kserve-container"
Jul 02 19:18:42.872045 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.871986    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerName="kserve-container"
Jul 02 19:18:42.872045 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.872043    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerName="kube-rbac-proxy"
Jul 02 19:18:42.872045 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.872050    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kserve-container"
Jul 02 19:18:42.872385 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.872056    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4efd3f46-02ad-4da3-bbb3-1e889542a4c4" containerName="kube-rbac-proxy"
Jul 02 19:18:42.872385 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.872062    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a1f430b-1817-4fd5-aeef-7d0b4b354bef" containerName="kserve-container"
Jul 02 19:18:42.874912 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.874896    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hw6zs/must-gather-d5f6t"
Jul 02 19:18:42.877919 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.877897    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hw6zs\"/\"openshift-service-ca.crt\""
Jul 02 19:18:42.878035 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.877904    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hw6zs\"/\"default-dockercfg-6msvh\""
Jul 02 19:18:42.879290 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.879276    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hw6zs\"/\"kube-root-ca.crt\""
Jul 02 19:18:42.882838 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.882817    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hw6zs/must-gather-d5f6t"]
Jul 02 19:18:42.886087 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.886064    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:18:42.953265 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.953229    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccrzl\" (UniqueName: \"kubernetes.io/projected/81fac94d-4973-4a2e-a50d-79c442a89290-kube-api-access-ccrzl\") pod \"must-gather-d5f6t\" (UID: \"81fac94d-4973-4a2e-a50d-79c442a89290\") " pod="openshift-must-gather-hw6zs/must-gather-d5f6t"
Jul 02 19:18:42.953441 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.953308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/81fac94d-4973-4a2e-a50d-79c442a89290-must-gather-output\") pod \"must-gather-d5f6t\" (UID: \"81fac94d-4973-4a2e-a50d-79c442a89290\") " pod="openshift-must-gather-hw6zs/must-gather-d5f6t"
Jul 02 19:18:42.986773 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:42.986742    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:18:43.054315 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:43.054283    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/81fac94d-4973-4a2e-a50d-79c442a89290-must-gather-output\") pod \"must-gather-d5f6t\" (UID: \"81fac94d-4973-4a2e-a50d-79c442a89290\") " pod="openshift-must-gather-hw6zs/must-gather-d5f6t"
Jul 02 19:18:43.054487 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:43.054349    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccrzl\" (UniqueName: \"kubernetes.io/projected/81fac94d-4973-4a2e-a50d-79c442a89290-kube-api-access-ccrzl\") pod \"must-gather-d5f6t\" (UID: \"81fac94d-4973-4a2e-a50d-79c442a89290\") " pod="openshift-must-gather-hw6zs/must-gather-d5f6t"
Jul 02 19:18:43.054628 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:43.054610    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/81fac94d-4973-4a2e-a50d-79c442a89290-must-gather-output\") pod \"must-gather-d5f6t\" (UID: \"81fac94d-4973-4a2e-a50d-79c442a89290\") " pod="openshift-must-gather-hw6zs/must-gather-d5f6t"
Jul 02 19:18:43.062558 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:43.062535    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccrzl\" (UniqueName: \"kubernetes.io/projected/81fac94d-4973-4a2e-a50d-79c442a89290-kube-api-access-ccrzl\") pod \"must-gather-d5f6t\" (UID: \"81fac94d-4973-4a2e-a50d-79c442a89290\") " pod="openshift-must-gather-hw6zs/must-gather-d5f6t"
Jul 02 19:18:43.202717 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:43.202618    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hw6zs/must-gather-d5f6t"
Jul 02 19:18:43.330236 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:43.330162    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hw6zs/must-gather-d5f6t"]
Jul 02 19:18:43.332656 ip-10-0-140-68 kubenswrapper[2575]: W0702 19:18:43.332624    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81fac94d_4973_4a2e_a50d_79c442a89290.slice/crio-9ce0fd09e87d8ea31c0bedf2dbbf41b4748df8faff6c738b6792b14ad9c393c0 WatchSource:0}: Error finding container 9ce0fd09e87d8ea31c0bedf2dbbf41b4748df8faff6c738b6792b14ad9c393c0: Status 404 returned error can't find the container with id 9ce0fd09e87d8ea31c0bedf2dbbf41b4748df8faff6c738b6792b14ad9c393c0
Jul 02 19:18:43.334519 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:43.334498    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 19:18:43.446068 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:43.446031    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hw6zs/must-gather-d5f6t" event={"ID":"81fac94d-4973-4a2e-a50d-79c442a89290","Type":"ContainerStarted","Data":"9ce0fd09e87d8ea31c0bedf2dbbf41b4748df8faff6c738b6792b14ad9c393c0"}
Jul 02 19:18:45.127224 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.127175    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:18:45.276736 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.276648    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj7bf\" (UniqueName: \"kubernetes.io/projected/f89b4832-6ab6-47ea-b907-00ea9a6d239b-kube-api-access-bj7bf\") pod \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\" (UID: \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\") "
Jul 02 19:18:45.276736 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.276700    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f89b4832-6ab6-47ea-b907-00ea9a6d239b-proxy-tls\") pod \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\" (UID: \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\") "
Jul 02 19:18:45.276964 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.276747    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f89b4832-6ab6-47ea-b907-00ea9a6d239b-kube-rbac-proxy-sar-config\") pod \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\" (UID: \"f89b4832-6ab6-47ea-b907-00ea9a6d239b\") "
Jul 02 19:18:45.277343 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.277309    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89b4832-6ab6-47ea-b907-00ea9a6d239b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f89b4832-6ab6-47ea-b907-00ea9a6d239b" (UID: "f89b4832-6ab6-47ea-b907-00ea9a6d239b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 19:18:45.279389 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.279249    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89b4832-6ab6-47ea-b907-00ea9a6d239b-kube-api-access-bj7bf" (OuterVolumeSpecName: "kube-api-access-bj7bf") pod "f89b4832-6ab6-47ea-b907-00ea9a6d239b" (UID: "f89b4832-6ab6-47ea-b907-00ea9a6d239b"). InnerVolumeSpecName "kube-api-access-bj7bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 19:18:45.279389 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.279270    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89b4832-6ab6-47ea-b907-00ea9a6d239b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f89b4832-6ab6-47ea-b907-00ea9a6d239b" (UID: "f89b4832-6ab6-47ea-b907-00ea9a6d239b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 19:18:45.378442 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.378405    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj7bf\" (UniqueName: \"kubernetes.io/projected/f89b4832-6ab6-47ea-b907-00ea9a6d239b-kube-api-access-bj7bf\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:18:45.378442 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.378443    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f89b4832-6ab6-47ea-b907-00ea9a6d239b-proxy-tls\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:18:45.378686 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.378462    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f89b4832-6ab6-47ea-b907-00ea9a6d239b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:18:45.458951 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.458906    2575 generic.go:356] "Generic (PLEG): container finished" podID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerID="9693586d46f8451a73bf503aabc9584373824cdfaa7a6f55df24617ca648cb3f" exitCode=0
Jul 02 19:18:45.459131 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.458960    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"
Jul 02 19:18:45.459131 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.459036    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" event={"ID":"f89b4832-6ab6-47ea-b907-00ea9a6d239b","Type":"ContainerDied","Data":"9693586d46f8451a73bf503aabc9584373824cdfaa7a6f55df24617ca648cb3f"}
Jul 02 19:18:45.459131 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.459098    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4" event={"ID":"f89b4832-6ab6-47ea-b907-00ea9a6d239b","Type":"ContainerDied","Data":"431f378f82d776329f571e6899b01296b5a3daa44b3d52f0589ab75858802445"}
Jul 02 19:18:45.459330 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.459139    2575 scope.go:117] "RemoveContainer" containerID="cd4e8089943da08010e0db8048a893ae1e4eb61cd3118c0983c5bd47376d4e02"
Jul 02 19:18:45.484648 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.484624    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"]
Jul 02 19:18:45.487554 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:45.487534    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-208dc-predictor-7dd8d55cdb-fm9m4"]
Jul 02 19:18:46.489246 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:46.489186    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" path="/var/lib/kubelet/pods/f89b4832-6ab6-47ea-b907-00ea9a6d239b/volumes"
Jul 02 19:18:48.409459 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:48.409264    2575 scope.go:117] "RemoveContainer" containerID="9693586d46f8451a73bf503aabc9584373824cdfaa7a6f55df24617ca648cb3f"
Jul 02 19:18:49.298940 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:49.298918    2575 scope.go:117] "RemoveContainer" containerID="cd4e8089943da08010e0db8048a893ae1e4eb61cd3118c0983c5bd47376d4e02"
Jul 02 19:18:49.299290 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:18:49.299263    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4e8089943da08010e0db8048a893ae1e4eb61cd3118c0983c5bd47376d4e02\": container with ID starting with cd4e8089943da08010e0db8048a893ae1e4eb61cd3118c0983c5bd47376d4e02 not found: ID does not exist" containerID="cd4e8089943da08010e0db8048a893ae1e4eb61cd3118c0983c5bd47376d4e02"
Jul 02 19:18:49.299334 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:49.299303    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4e8089943da08010e0db8048a893ae1e4eb61cd3118c0983c5bd47376d4e02"} err="failed to get container status \"cd4e8089943da08010e0db8048a893ae1e4eb61cd3118c0983c5bd47376d4e02\": rpc error: code = NotFound desc = could not find container \"cd4e8089943da08010e0db8048a893ae1e4eb61cd3118c0983c5bd47376d4e02\": container with ID starting with cd4e8089943da08010e0db8048a893ae1e4eb61cd3118c0983c5bd47376d4e02 not found: ID does not exist"
Jul 02 19:18:49.299334 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:49.299324    2575 scope.go:117] "RemoveContainer" containerID="9693586d46f8451a73bf503aabc9584373824cdfaa7a6f55df24617ca648cb3f"
Jul 02 19:18:49.299608 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:18:49.299593    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9693586d46f8451a73bf503aabc9584373824cdfaa7a6f55df24617ca648cb3f\": container with ID starting with 9693586d46f8451a73bf503aabc9584373824cdfaa7a6f55df24617ca648cb3f not found: ID does not exist" containerID="9693586d46f8451a73bf503aabc9584373824cdfaa7a6f55df24617ca648cb3f"
Jul 02 19:18:49.299649 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:49.299614    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9693586d46f8451a73bf503aabc9584373824cdfaa7a6f55df24617ca648cb3f"} err="failed to get container status \"9693586d46f8451a73bf503aabc9584373824cdfaa7a6f55df24617ca648cb3f\": rpc error: code = NotFound desc = could not find container \"9693586d46f8451a73bf503aabc9584373824cdfaa7a6f55df24617ca648cb3f\": container with ID starting with 9693586d46f8451a73bf503aabc9584373824cdfaa7a6f55df24617ca648cb3f not found: ID does not exist"
Jul 02 19:18:50.493761 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:50.493717    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hw6zs/must-gather-d5f6t" event={"ID":"81fac94d-4973-4a2e-a50d-79c442a89290","Type":"ContainerStarted","Data":"71365f047f0df775f0f403b8ad284ffbeffbc9ecedc65abb22306acdf2ce9cfa"}
Jul 02 19:18:50.494281 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:50.493773    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hw6zs/must-gather-d5f6t" event={"ID":"81fac94d-4973-4a2e-a50d-79c442a89290","Type":"ContainerStarted","Data":"f1c54319758cdb005b49cb0cc344f3361bab409b9c6d2d1a4be139975c6c97eb"}
Jul 02 19:18:50.510409 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:18:50.510356    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hw6zs/must-gather-d5f6t" podStartSLOduration=2.436051444 podStartE2EDuration="8.510338754s" totalImagesPullingTime="6.07428731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:18:50.509072249 +0000 UTC m=+2922.578699777" watchObservedRunningTime="2026-07-02 19:18:50.510338754 +0000 UTC m=+2922.579966411"
Jul 02 19:19:11.582367 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:11.582335    2575 generic.go:356] "Generic (PLEG): container finished" podID="81fac94d-4973-4a2e-a50d-79c442a89290" containerID="f1c54319758cdb005b49cb0cc344f3361bab409b9c6d2d1a4be139975c6c97eb" exitCode=0
Jul 02 19:19:11.582794 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:11.582406    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hw6zs/must-gather-d5f6t" event={"ID":"81fac94d-4973-4a2e-a50d-79c442a89290","Type":"ContainerDied","Data":"f1c54319758cdb005b49cb0cc344f3361bab409b9c6d2d1a4be139975c6c97eb"}
Jul 02 19:19:11.582794 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:11.582701    2575 scope.go:117] "RemoveContainer" containerID="f1c54319758cdb005b49cb0cc344f3361bab409b9c6d2d1a4be139975c6c97eb"
Jul 02 19:19:12.466723 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:12.466687    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hw6zs_must-gather-d5f6t_81fac94d-4973-4a2e-a50d-79c442a89290/gather/0.log"
Jul 02 19:19:12.924514 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:12.924483    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:19:13.082553 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:13.082521    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:19:13.141038 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:13.141005    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:19:15.656519 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:15.656487    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5j4fb_d3682ee4-82ef-4457-a048-4de940c25ac0/global-pull-secret-syncer/0.log"
Jul 02 19:19:15.792003 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:15.791973    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log"
Jul 02 19:19:15.972431 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:15.972351    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-68.ec2.internal_000fc49438e162666c8381fedce28b6d/haproxy/0.log"
Jul 02 19:19:18.001449 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.001410    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-hw6zs/must-gather-d5f6t"]
Jul 02 19:19:18.001901 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.001584    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-hw6zs/must-gather-d5f6t" podUID="81fac94d-4973-4a2e-a50d-79c442a89290" containerName="copy" containerID="cri-o://71365f047f0df775f0f403b8ad284ffbeffbc9ecedc65abb22306acdf2ce9cfa" gracePeriod=2
Jul 02 19:19:18.004001 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.003978    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-hw6zs/must-gather-d5f6t"]
Jul 02 19:19:18.004411 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:19:18.004380    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-d5f6t\" is forbidden: User \"system:node:ip-10-0-140-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hw6zs\": no relationship found between node 'ip-10-0-140-68.ec2.internal' and this object" podUID="81fac94d-4973-4a2e-a50d-79c442a89290" pod="openshift-must-gather-hw6zs/must-gather-d5f6t"
Jul 02 19:19:18.228831 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.228808    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hw6zs_must-gather-d5f6t_81fac94d-4973-4a2e-a50d-79c442a89290/copy/0.log"
Jul 02 19:19:18.230402 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.230385    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hw6zs/must-gather-d5f6t"
Jul 02 19:19:18.232808 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:19:18.232782    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-d5f6t\" is forbidden: User \"system:node:ip-10-0-140-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hw6zs\": no relationship found between node 'ip-10-0-140-68.ec2.internal' and this object" podUID="81fac94d-4973-4a2e-a50d-79c442a89290" pod="openshift-must-gather-hw6zs/must-gather-d5f6t"
Jul 02 19:19:18.260420 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.260359    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/81fac94d-4973-4a2e-a50d-79c442a89290-must-gather-output\") pod \"81fac94d-4973-4a2e-a50d-79c442a89290\" (UID: \"81fac94d-4973-4a2e-a50d-79c442a89290\") "
Jul 02 19:19:18.260420 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.260411    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccrzl\" (UniqueName: \"kubernetes.io/projected/81fac94d-4973-4a2e-a50d-79c442a89290-kube-api-access-ccrzl\") pod \"81fac94d-4973-4a2e-a50d-79c442a89290\" (UID: \"81fac94d-4973-4a2e-a50d-79c442a89290\") "
Jul 02 19:19:18.262498 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.262465    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fac94d-4973-4a2e-a50d-79c442a89290-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "81fac94d-4973-4a2e-a50d-79c442a89290" (UID: "81fac94d-4973-4a2e-a50d-79c442a89290"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 19:19:18.262604 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.262537    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fac94d-4973-4a2e-a50d-79c442a89290-kube-api-access-ccrzl" (OuterVolumeSpecName: "kube-api-access-ccrzl") pod "81fac94d-4973-4a2e-a50d-79c442a89290" (UID: "81fac94d-4973-4a2e-a50d-79c442a89290"). InnerVolumeSpecName "kube-api-access-ccrzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 19:19:18.361199 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.361166    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccrzl\" (UniqueName: \"kubernetes.io/projected/81fac94d-4973-4a2e-a50d-79c442a89290-kube-api-access-ccrzl\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:19:18.361357 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.361214    2575 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/81fac94d-4973-4a2e-a50d-79c442a89290-must-gather-output\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Jul 02 19:19:18.487485 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:19:18.487451    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-d5f6t\" is forbidden: User \"system:node:ip-10-0-140-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hw6zs\": no relationship found between node 'ip-10-0-140-68.ec2.internal' and this object" podUID="81fac94d-4973-4a2e-a50d-79c442a89290" pod="openshift-must-gather-hw6zs/must-gather-d5f6t"
Jul 02 19:19:18.488096 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.488075    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81fac94d-4973-4a2e-a50d-79c442a89290" path="/var/lib/kubelet/pods/81fac94d-4973-4a2e-a50d-79c442a89290/volumes"
Jul 02 19:19:18.505147 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.505115    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:19:18.562284 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.562254    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wdxk_72fc1e32-b32e-415f-9101-62498e24ea7b/konnectivity-agent/0.log" limit=1024
Jul 02 19:19:18.609484 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.609458    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hw6zs_must-gather-d5f6t_81fac94d-4973-4a2e-a50d-79c442a89290/copy/0.log"
Jul 02 19:19:18.609832 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.609807    2575 generic.go:356] "Generic (PLEG): container finished" podID="81fac94d-4973-4a2e-a50d-79c442a89290" containerID="71365f047f0df775f0f403b8ad284ffbeffbc9ecedc65abb22306acdf2ce9cfa" exitCode=143
Jul 02 19:19:18.609968 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.609864    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hw6zs/must-gather-d5f6t"
Jul 02 19:19:18.609968 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.609898    2575 scope.go:117] "RemoveContainer" containerID="71365f047f0df775f0f403b8ad284ffbeffbc9ecedc65abb22306acdf2ce9cfa"
Jul 02 19:19:18.617434 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.617412    2575 scope.go:117] "RemoveContainer" containerID="f1c54319758cdb005b49cb0cc344f3361bab409b9c6d2d1a4be139975c6c97eb"
Jul 02 19:19:18.629974 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.629952    2575 scope.go:117] "RemoveContainer" containerID="71365f047f0df775f0f403b8ad284ffbeffbc9ecedc65abb22306acdf2ce9cfa"
Jul 02 19:19:18.630316 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:19:18.630278    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71365f047f0df775f0f403b8ad284ffbeffbc9ecedc65abb22306acdf2ce9cfa\": container with ID starting with 71365f047f0df775f0f403b8ad284ffbeffbc9ecedc65abb22306acdf2ce9cfa not found: ID does not exist" containerID="71365f047f0df775f0f403b8ad284ffbeffbc9ecedc65abb22306acdf2ce9cfa"
Jul 02 19:19:18.630464 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.630325    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71365f047f0df775f0f403b8ad284ffbeffbc9ecedc65abb22306acdf2ce9cfa"} err="failed to get container status \"71365f047f0df775f0f403b8ad284ffbeffbc9ecedc65abb22306acdf2ce9cfa\": rpc error: code = NotFound desc = could not find container \"71365f047f0df775f0f403b8ad284ffbeffbc9ecedc65abb22306acdf2ce9cfa\": container with ID starting with 71365f047f0df775f0f403b8ad284ffbeffbc9ecedc65abb22306acdf2ce9cfa not found: ID does not exist"
Jul 02 19:19:18.630464 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.630344    2575 scope.go:117] "RemoveContainer" containerID="f1c54319758cdb005b49cb0cc344f3361bab409b9c6d2d1a4be139975c6c97eb"
Jul 02 19:19:18.630772 ip-10-0-140-68 kubenswrapper[2575]: E0702 19:19:18.630740    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c54319758cdb005b49cb0cc344f3361bab409b9c6d2d1a4be139975c6c97eb\": container with ID starting with f1c54319758cdb005b49cb0cc344f3361bab409b9c6d2d1a4be139975c6c97eb not found: ID does not exist" containerID="f1c54319758cdb005b49cb0cc344f3361bab409b9c6d2d1a4be139975c6c97eb"
Jul 02 19:19:18.630859 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:18.630782    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c54319758cdb005b49cb0cc344f3361bab409b9c6d2d1a4be139975c6c97eb"} err="failed to get container status \"f1c54319758cdb005b49cb0cc344f3361bab409b9c6d2d1a4be139975c6c97eb\": rpc error: code = NotFound desc = could not find container \"f1c54319758cdb005b49cb0cc344f3361bab409b9c6d2d1a4be139975c6c97eb\": container with ID starting with f1c54319758cdb005b49cb0cc344f3361bab409b9c6d2d1a4be139975c6c97eb not found: ID does not exist"
Jul 02 19:19:20.383900 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:20.383871    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-gstlk_5b2c35b5-1bcf-4674-9b53-60abc6db4c07/cluster-monitoring-operator/0.log"
Jul 02 19:19:20.786918 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:20.786886    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gt7jn_b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95/node-exporter/0.log"
Jul 02 19:19:20.808964 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:20.808937    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gt7jn_b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95/kube-rbac-proxy/0.log"
Jul 02 19:19:20.833692 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:20.833665    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gt7jn_b9b64b2d-5cef-4ba4-b19e-d2471cdf4f95/init-textfile/0.log"
Jul 02 19:19:20.961983 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:20.961907    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b19f68e7-3fe0-42c8-9487-e90ac8e68e26/prometheus/0.log"
Jul 02 19:19:20.981261 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:20.981237    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b19f68e7-3fe0-42c8-9487-e90ac8e68e26/config-reloader/0.log"
Jul 02 19:19:21.004238 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:21.004208    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b19f68e7-3fe0-42c8-9487-e90ac8e68e26/thanos-sidecar/0.log"
Jul 02 19:19:21.029992 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:21.029948    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b19f68e7-3fe0-42c8-9487-e90ac8e68e26/kube-rbac-proxy-web/0.log"
Jul 02 19:19:21.054208 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:21.054168    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b19f68e7-3fe0-42c8-9487-e90ac8e68e26/kube-rbac-proxy/0.log"
Jul 02 19:19:21.076329 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:21.076291    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b19f68e7-3fe0-42c8-9487-e90ac8e68e26/kube-rbac-proxy-thanos/0.log"
Jul 02 19:19:21.100501 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:21.100482    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b19f68e7-3fe0-42c8-9487-e90ac8e68e26/init-config-reloader/0.log"
Jul 02 19:19:21.177652 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:21.177625    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-fd8v2_253f9a92-7d58-48c5-be09-36c69d2559e7/prometheus-operator-admission-webhook/0.log"
Jul 02 19:19:23.228165 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:23.228137    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/1.log"
Jul 02 19:19:23.232872 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:23.232854    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7d9mj_f4e75b84-1c82-46c8-b414-f252d7ac5633/console-operator/2.log"
Jul 02 19:19:24.164609 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.164583    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-pqt6r_becbcfb3-b9cc-409a-8a1c-f70ed6cb7b72/volume-data-source-validator/0.log"
Jul 02 19:19:24.340572 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.340541    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"]
Jul 02 19:19:24.341057 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.340950    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerName="kserve-container"
Jul 02 19:19:24.341057 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.340968    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerName="kserve-container"
Jul 02 19:19:24.341057 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.340987    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81fac94d-4973-4a2e-a50d-79c442a89290" containerName="gather"
Jul 02 19:19:24.341057 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.340996    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="81fac94d-4973-4a2e-a50d-79c442a89290" containerName="gather"
Jul 02 19:19:24.341057 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.341015    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81fac94d-4973-4a2e-a50d-79c442a89290" containerName="copy"
Jul 02 19:19:24.341057 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.341024    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="81fac94d-4973-4a2e-a50d-79c442a89290" containerName="copy"
Jul 02 19:19:24.341057 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.341045    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerName="kube-rbac-proxy"
Jul 02 19:19:24.341057 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.341052    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerName="kube-rbac-proxy"
Jul 02 19:19:24.341484 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.341124    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerName="kserve-container"
Jul 02 19:19:24.341484 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.341137    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="81fac94d-4973-4a2e-a50d-79c442a89290" containerName="gather"
Jul 02 19:19:24.341484 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.341146    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f89b4832-6ab6-47ea-b907-00ea9a6d239b" containerName="kube-rbac-proxy"
Jul 02 19:19:24.341484 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.341156    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="81fac94d-4973-4a2e-a50d-79c442a89290" containerName="copy"
Jul 02 19:19:24.343148 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.343128    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.346040 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.346024    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xqmdr\"/\"openshift-service-ca.crt\""
Jul 02 19:19:24.347402 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.347382    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xqmdr\"/\"default-dockercfg-k4zn2\""
Jul 02 19:19:24.347490 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.347404    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xqmdr\"/\"kube-root-ca.crt\""
Jul 02 19:19:24.353527 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.353507    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"]
Jul 02 19:19:24.407546 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.407509    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7a0a516-78f1-42fd-b321-8a8876aad037-podres\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.407725 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.407568    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w7cp\" (UniqueName: \"kubernetes.io/projected/b7a0a516-78f1-42fd-b321-8a8876aad037-kube-api-access-7w7cp\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.407725 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.407626    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7a0a516-78f1-42fd-b321-8a8876aad037-sys\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.407725 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.407641    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7a0a516-78f1-42fd-b321-8a8876aad037-proc\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.407725 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.407680    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7a0a516-78f1-42fd-b321-8a8876aad037-lib-modules\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.509124 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.509035    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7cp\" (UniqueName: \"kubernetes.io/projected/b7a0a516-78f1-42fd-b321-8a8876aad037-kube-api-access-7w7cp\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.509124 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.509089    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7a0a516-78f1-42fd-b321-8a8876aad037-sys\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.509385 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.509157    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7a0a516-78f1-42fd-b321-8a8876aad037-sys\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.509385 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.509215    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7a0a516-78f1-42fd-b321-8a8876aad037-proc\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.509385 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.509244    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7a0a516-78f1-42fd-b321-8a8876aad037-lib-modules\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.509385 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.509293    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7a0a516-78f1-42fd-b321-8a8876aad037-podres\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.509385 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.509324    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7a0a516-78f1-42fd-b321-8a8876aad037-proc\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.509587 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.509403    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7a0a516-78f1-42fd-b321-8a8876aad037-podres\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.509587 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.509423    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7a0a516-78f1-42fd-b321-8a8876aad037-lib-modules\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.517438 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.517411    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7cp\" (UniqueName: \"kubernetes.io/projected/b7a0a516-78f1-42fd-b321-8a8876aad037-kube-api-access-7w7cp\") pod \"perf-node-gather-daemonset-jf8r5\" (UID: \"b7a0a516-78f1-42fd-b321-8a8876aad037\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.652792 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.652760    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:24.779121 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:24.779100    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"]
Jul 02 19:19:24.781208 ip-10-0-140-68 kubenswrapper[2575]: W0702 19:19:24.781167    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb7a0a516_78f1_42fd_b321_8a8876aad037.slice/crio-c8153daa7c17bc835da2ea0ba797f1989bb15d58986cb06ce84025bcfbcee02e WatchSource:0}: Error finding container c8153daa7c17bc835da2ea0ba797f1989bb15d58986cb06ce84025bcfbcee02e: Status 404 returned error can't find the container with id c8153daa7c17bc835da2ea0ba797f1989bb15d58986cb06ce84025bcfbcee02e
Jul 02 19:19:25.146465 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:25.146439    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9zq5n_684a7ad0-005b-49d6-817b-b5bc7cbfb719/dns/0.log"
Jul 02 19:19:25.172326 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:25.172286    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9zq5n_684a7ad0-005b-49d6-817b-b5bc7cbfb719/kube-rbac-proxy/0.log"
Jul 02 19:19:25.270588 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:25.270558    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6mtkp_d5a96438-0081-4f62-aa03-cd7cf1bb0f2c/dns-node-resolver/0.log"
Jul 02 19:19:25.639078 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:25.639044    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5" event={"ID":"b7a0a516-78f1-42fd-b321-8a8876aad037","Type":"ContainerStarted","Data":"f6566dd9bf8c64e81f8842778c6f5fd74f59049afa5e218c74bce20f2a808be1"}
Jul 02 19:19:25.639607 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:25.639117    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5" event={"ID":"b7a0a516-78f1-42fd-b321-8a8876aad037","Type":"ContainerStarted","Data":"c8153daa7c17bc835da2ea0ba797f1989bb15d58986cb06ce84025bcfbcee02e"}
Jul 02 19:19:25.639607 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:25.639214    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:25.655335 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:25.655288    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5" podStartSLOduration=1.655274718 podStartE2EDuration="1.655274718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:19:25.654441455 +0000 UTC m=+2957.724068985" watchObservedRunningTime="2026-07-02 19:19:25.655274718 +0000 UTC m=+2957.724902245"
Jul 02 19:19:25.781333 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:25.781306    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6cwjc_02cf5868-e870-47eb-870f-c87dcabdae5a/node-ca/0.log"
Jul 02 19:19:26.594751 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:26.594726    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f6f776dd5-qrrs8_b7d790ff-691d-4fde-81c7-613493010723/router/0.log"
Jul 02 19:19:27.031422 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:27.031322    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qt4zw_6fa3f945-1298-44c4-8ec7-33014cddcd23/serve-healthcheck-canary/0.log"
Jul 02 19:19:27.369011 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:27.368979    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-w5w95_6846ce7e-056b-4aad-88cd-572cf4d80df3/insights-operator/0.log"
Jul 02 19:19:27.369633 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:27.369614    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-w5w95_6846ce7e-056b-4aad-88cd-572cf4d80df3/insights-operator/1.log"
Jul 02 19:19:27.523643 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:27.523601    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dp4vg_572ae32e-0565-40c9-97eb-21161b4fb2c9/kube-rbac-proxy/0.log"
Jul 02 19:19:27.545176 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:27.545152    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dp4vg_572ae32e-0565-40c9-97eb-21161b4fb2c9/exporter/0.log"
Jul 02 19:19:27.567268 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:27.567246    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dp4vg_572ae32e-0565-40c9-97eb-21161b4fb2c9/extractor/0.log"
Jul 02 19:19:29.793315 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:29.793276    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-458lw_16c37bff-72fe-4e97-b475-8d6beb710aac/s3-init/0.log"
Jul 02 19:19:29.816806 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:29.816777    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-458lw_16c37bff-72fe-4e97-b475-8d6beb710aac/download-hf-model/0.log"
Jul 02 19:19:29.860451 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:29.860426    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-j5hhk_e6fd7045-82f3-408e-95c0-9b5445775044/seaweedfs/0.log"
Jul 02 19:19:31.652358 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:31.652290    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-jf8r5"
Jul 02 19:19:34.036451 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:34.036413    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-8bvjs_cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f/kube-storage-version-migrator-operator/1.log"
Jul 02 19:19:34.037163 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:34.037144    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-8bvjs_cdb708ea-5ecd-4a2c-a4d4-08565fe50f2f/kube-storage-version-migrator-operator/0.log"
Jul 02 19:19:35.096545 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:35.096519    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6f6sb_98e2289a-1242-48fa-9b3e-a89b46c2c9d8/kube-multus-additional-cni-plugins/0.log"
Jul 02 19:19:35.120425 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:35.120395    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6f6sb_98e2289a-1242-48fa-9b3e-a89b46c2c9d8/egress-router-binary-copy/0.log"
Jul 02 19:19:35.143024 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:35.142998    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6f6sb_98e2289a-1242-48fa-9b3e-a89b46c2c9d8/cni-plugins/0.log"
Jul 02 19:19:35.167821 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:35.167798    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6f6sb_98e2289a-1242-48fa-9b3e-a89b46c2c9d8/bond-cni-plugin/0.log"
Jul 02 19:19:35.195356 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:35.195336    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6f6sb_98e2289a-1242-48fa-9b3e-a89b46c2c9d8/routeoverride-cni/0.log"
Jul 02 19:19:35.222979 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:35.222953    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6f6sb_98e2289a-1242-48fa-9b3e-a89b46c2c9d8/whereabouts-cni-bincopy/0.log"
Jul 02 19:19:35.252802 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:35.252735    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6f6sb_98e2289a-1242-48fa-9b3e-a89b46c2c9d8/whereabouts-cni/0.log"
Jul 02 19:19:35.650026 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:35.649994    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rglzr_e84ffa74-4554-441b-8e99-be4646050843/kube-multus/0.log"
Jul 02 19:19:35.714236 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:35.714209    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m2h4c_780da6ef-b5e6-43c9-b539-185cc5434c63/network-metrics-daemon/0.log"
Jul 02 19:19:35.735425 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:35.735379    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m2h4c_780da6ef-b5e6-43c9-b539-185cc5434c63/kube-rbac-proxy/0.log"
Jul 02 19:19:36.960628 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:36.960602    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-controller/0.log"
Jul 02 19:19:36.982080 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:36.982059    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/1.log"
Jul 02 19:19:36.982218 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:36.982164    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovn-acl-logging/0.log"
Jul 02 19:19:37.001605 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:37.001582    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/kube-rbac-proxy-node/0.log"
Jul 02 19:19:37.026093 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:37.026068    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 19:19:37.054055 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:37.054030    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/northd/0.log"
Jul 02 19:19:37.077552 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:37.077526    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/nbdb/0.log"
Jul 02 19:19:37.099629 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:37.099604    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/sbdb/0.log"
Jul 02 19:19:37.149930 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:37.149901    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkdd4_0e4b9b59-08ac-4d0b-a98c-48b6a4881c2b/ovnkube-controller/0.log"
Jul 02 19:19:38.147097 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:38.147055    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-b7jnx_7340eb6c-cda8-42dc-9abe-2b16be84f537/network-check-target-container/0.log"
Jul 02 19:19:39.118674 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:39.118643    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-m8sht_f1197266-26d4-4a4d-9652-0db22827bf3b/iptables-alerter/0.log"
Jul 02 19:19:39.702369 ip-10-0-140-68 kubenswrapper[2575]: I0702 19:19:39.702339    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-hhvg5_c01f69e9-cff9-4c5a-97d0-68a53d05b20b/tuned/0.log"