Jul 02 13:12:09.920720 ip-10-0-131-243 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 02 13:12:09.920731 ip-10-0-131-243 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 02 13:12:09.920738 ip-10-0-131-243 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 02 13:12:09.921048 ip-10-0-131-243 systemd[1]: Failed to start Kubernetes Kubelet. Jul 02 13:12:20.734762 ip-10-0-131-243 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 02 13:12:20.734776 ip-10-0-131-243 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a6812e5f5acd4281ae7f5c5759b06c8c -- Jul 02 13:13:55.039148 ip-10-0-131-243 systemd[1]: Starting Kubernetes Kubelet... Jul 02 13:13:55.508190 ip-10-0-131-243 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 13:13:55.508190 ip-10-0-131-243 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 02 13:13:55.508190 ip-10-0-131-243 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 13:13:55.508190 ip-10-0-131-243 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 02 13:13:55.508190 ip-10-0-131-243 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 13:13:55.508190 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.441907 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 02 13:13:55.508190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.447991 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 13:13:55.508190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448001 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 13:13:55.508190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448005 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 13:13:55.508190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448008 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 13:13:55.508190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448010 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 13:13:55.508190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448014 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 13:13:55.508190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448017 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 13:13:55.508190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448020 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448023 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448026 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448029 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448032 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448034 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448037 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448039 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448042 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448045 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448047 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448050 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448053 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448055 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448058 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448060 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448063 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448066 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448068 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 13:13:55.515200 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448071 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448074 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448076 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448079 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448083 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448087 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448089 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448092 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448095 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448097 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448100 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448102 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448105 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448109 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448113 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448115 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448118 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448121 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448124 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448127 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 13:13:55.515814 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448129 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448132 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448135 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448137 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448140 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448142 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448145 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448147 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448150 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448152 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448162 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448166 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448168 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448171 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448174 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448177 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448179 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448183 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448185 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 13:13:55.516390 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448188 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448191 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448193 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448196 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448199 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448201 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448204 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448207 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448210 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448212 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448215 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448217 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448220 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448223 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448226 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448229 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448231 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448234 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448236 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448239 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 13:13:55.516940 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448242 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448244 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448247 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448249 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448253 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448257 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448261 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448264 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448266 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448270 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448272 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448275 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448278 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.448280 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451598 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451609 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451613 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451616 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451619 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 13:13:55.517473 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451622 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451627 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451630 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451633 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451636 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451639 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451642 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451646 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451650 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451653 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451656 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451658 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451661 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451664 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451666 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451670 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451672 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451676 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451679 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 13:13:55.518190 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451681 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451684 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451687 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451689 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451693 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451695 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451698 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451700 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451704 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451706 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451709 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451712 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451715 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451717 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451720 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451722 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451725 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451727 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451730 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 13:13:55.518829 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451733 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451736 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451738 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451741 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451743 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451746 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451748 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451751 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451754 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451756 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451759 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451762 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451764 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451767 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451770 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451772 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451775 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451777 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451780 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451783 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 13:13:55.519471 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451787 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451790 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451793 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451796 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451798 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451801 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451803 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451806 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451808 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451811 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451814 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451817 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451819 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451822 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451824 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451827 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451830 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451832 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451835 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451837 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 13:13:55.520104 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451841 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451845 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451847 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451850 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451852 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451855 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451858 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451860 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451862 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451865 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451868 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451871 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451873 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451876 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451878 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.451881 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452278 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452286 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452289 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452292 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 13:13:55.520713 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452295 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452298 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452301 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452303 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452306 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452309 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452311 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452314 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452316 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452319 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452321 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452325 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452336 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452341 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452344 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452346 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452349 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452352 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452355 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 13:13:55.521338 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452357 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452360 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452362 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452365 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452368 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452370 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452373 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452375 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452377 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452381 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452384 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452387 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452390 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452393 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452396 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452399 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452402 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452405 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452407 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 13:13:55.522184 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452410 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452412 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452415 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452418 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452420 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452422 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452426 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452431 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452433 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452437 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452439 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452442 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452444 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452447 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452449 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452452 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452454 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452457 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452459 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452461 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 13:13:55.522780 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452464 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452466 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452469 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452471 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452474 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452476 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452479 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452481 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452483 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452486 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452488 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452490 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452493 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452495 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452498 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452500 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452502 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452505 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452508 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452512 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 13:13:55.523378 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452515 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452518 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452520 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452523 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452525 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452527 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452530 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452533 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452535 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452537 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452540 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452542 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452545 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452547 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452550 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452552 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.452555 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452617 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452623 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452629 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452633 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 02 13:13:55.524008 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452637 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452640 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452644 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452648 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452654 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452657 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452660 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452663 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452666 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452671 2578 flags.go:64] FLAG: --cgroup-root="" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452675 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452678 2578 flags.go:64] FLAG: --client-ca-file="" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452682 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452685 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452689 2578 flags.go:64] FLAG: --cluster-domain="" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452692 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452695 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452699 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452702 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452706 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452709 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452711 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452715 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452718 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 02 13:13:55.524576 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452720 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452723 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452726 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452729 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452734 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452737 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452739 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452742 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452745 2578 flags.go:64] FLAG: --enable-server="true" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452748 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452751 2578 flags.go:64] FLAG: --event-burst="100" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452754 2578 flags.go:64] FLAG: --event-qps="50" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452757 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452760 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452763 2578 flags.go:64] FLAG: --eviction-hard="" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452766 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452769 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452775 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452780 2578 flags.go:64] FLAG: --eviction-soft="" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452783 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452786 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452790 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452793 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452796 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452798 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 02 13:13:55.658380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452801 2578 flags.go:64] FLAG: --feature-gates="" Jul 02 13:13:55.529511 ip-10-0-131-243 systemd[1]: Started Kubernetes Kubelet. Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452805 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452808 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452811 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452814 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452816 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452819 2578 flags.go:64] FLAG: --help="false" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452822 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452825 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452828 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452831 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452834 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452837 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452840 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452843 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452846 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452848 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452851 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452854 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452857 2578 flags.go:64] FLAG: --kube-reserved="" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452860 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452862 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452865 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452868 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 02 13:13:55.683725 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452872 2578 flags.go:64] FLAG: --lock-file="" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452876 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452879 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452882 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452887 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452890 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452893 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452896 2578 flags.go:64] FLAG: --logging-format="text" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452899 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452902 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452905 2578 flags.go:64] FLAG: --manifest-url="" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452908 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452927 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452932 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452936 2578 flags.go:64] FLAG: --max-pods="110" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452939 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452942 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452945 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452948 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452951 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452953 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452956 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452963 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452966 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452969 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 02 13:13:55.685351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452972 2578 flags.go:64] FLAG: --pod-cidr="" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452974 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452980 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452982 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452985 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452988 2578 flags.go:64] FLAG: --port="10250" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452991 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.452994 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c45a1a056085b07d" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453000 2578 flags.go:64] FLAG: --qos-reserved="" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453003 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453005 2578 flags.go:64] FLAG: --register-node="true" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453008 2578 flags.go:64] FLAG: --register-with-taints="" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453012 2578 flags.go:64] FLAG: --registry-burst="10" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453015 2578 flags.go:64] FLAG: --registry-qps="5" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453018 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453020 2578 flags.go:64] FLAG: --reserved-memory="" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453024 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453027 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453030 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453033 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453035 2578 flags.go:64] FLAG: --runonce="false" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453038 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453041 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453044 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453046 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 02 13:13:55.686690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453049 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453052 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453055 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453058 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453061 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453064 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453067 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453069 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453072 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453075 2578 flags.go:64] FLAG: --system-cgroups="" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453078 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453083 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453085 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453088 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453093 2578 flags.go:64] FLAG: --tls-min-version="" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453098 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453101 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453103 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453106 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453109 2578 flags.go:64] FLAG: --v="2" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453113 2578 flags.go:64] FLAG: --version="false" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453117 2578 flags.go:64] FLAG: --vmodule="" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453121 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.453124 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453239 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 13:13:55.689820 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453243 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453246 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453249 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453252 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453254 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453257 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453260 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453262 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453265 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453267 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453270 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453272 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453275 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453277 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453280 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453282 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453285 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453287 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453290 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 13:13:55.690711 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453292 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453294 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453297 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453302 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453304 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453307 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453310 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453312 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453314 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453317 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453320 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453323 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453325 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453328 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453330 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453332 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453335 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453338 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453342 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 13:13:55.696666 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453345 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453348 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453351 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453354 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453356 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453359 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453361 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453364 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453366 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453369 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453372 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453374 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453377 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453379 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453382 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453384 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453390 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453394 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453397 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453400 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 13:13:55.697382 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453403 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453405 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453408 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453411 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453414 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453417 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453419 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453422 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453425 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453427 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453430 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453433 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453436 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453439 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453441 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453444 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453446 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453449 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453452 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453454 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 13:13:55.698064 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453457 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453459 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453462 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453464 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453466 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453469 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453472 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453474 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453486 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453489 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453491 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453494 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453498 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453502 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453505 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453508 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453510 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453514 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453516 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 13:13:55.698794 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.453519 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.454093 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.460463 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.460478 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.460496 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.460501 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460554 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460560 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460565 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460568 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460572 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460574 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460577 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460580 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460583 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 13:13:55.699674 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460586 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460588 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460591 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460593 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460596 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460599 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460601 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460604 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460608 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460612 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460615 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460618 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460621 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460624 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460627 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460631 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460634 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460637 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460640 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 13:13:55.700336 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460643 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460646 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460648 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460651 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460654 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460657 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460660 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460663 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460665 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460668 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460671 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460677 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460680 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460683 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460685 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460688 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460691 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460693 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460696 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460699 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 13:13:55.701074 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460701 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460704 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460706 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460709 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460713 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460715 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460718 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460721 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460723 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460726 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460729 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460731 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460734 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460736 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460739 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460741 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460744 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460747 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460749 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 13:13:55.701912 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460751 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460754 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460756 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460759 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460763 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460765 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460768 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460770 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460773 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460776 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460778 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460781 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460783 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460786 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460789 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460792 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460794 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460798 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460801 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 13:13:55.702619 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460804 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460807 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460810 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460813 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460816 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460819 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460822 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460825 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460827 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460830 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460833 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460835 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460838 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.460842 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460971 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 13:13:55.703280 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460977 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460979 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460983 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460986 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460988 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460991 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460993 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460996 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.460999 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461002 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461004 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461007 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461010 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461012 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461015 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461018 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461022 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461024 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461027 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461029 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 13:13:55.703768 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461032 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461034 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461038 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461040 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461043 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461045 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461048 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461050 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461053 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461056 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461060 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461063 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461065 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461068 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461072 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461074 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461077 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461080 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461082 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461084 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 13:13:55.704410 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461087 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461089 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461092 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461096 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461099 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461102 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461105 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461107 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461112 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461115 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461117 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461120 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461123 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461125 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461128 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461131 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461133 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461136 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461139 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 13:13:55.704979 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461141 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461143 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461146 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461149 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461151 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461153 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461156 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461160 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461162 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461165 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461167 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461170 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461173 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461175 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461178 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461180 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461183 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461185 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461188 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461190 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 13:13:55.705567 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461192 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461197 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461199 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461202 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461204 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461207 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461209 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461212 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461215 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461223 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461226 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461229 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461231 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461234 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461236 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461239 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461241 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461244 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 13:13:55.878379 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:55.461246 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 13:13:55.879156 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.461255 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 13:13:55.879156 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.461345 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 02 13:13:55.879156 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.463256 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 02 13:13:55.879156 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.464607 2578 server.go:1019] "Starting client certificate rotation" Jul 02 13:13:55.879156 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.464710 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 13:13:55.879156 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.464747 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 13:13:55.879156 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.467095 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 13:13:55.879156 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.468152 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 13:13:55.879156 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.478626 2578 log.go:25] "Validated CRI v1 runtime API" Jul 02 13:13:55.879156 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.485139 2578 log.go:25] "Validated CRI v1 image API" Jul 02 13:13:55.879156 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.485880 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 02 13:13:55.879156 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.488189 2578 fs.go:135] Filesystem UUIDs: map[175732d9-04f3-43a0-a270-7a7a9a5909f3:/dev/nvme0n1p3 2117a93c-2d2f-475f-9449-cc5cbb21f8cf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 02 13:13:55.879656 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.488205 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 02 13:13:55.879770 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.494044 2578 manager.go:216] Machine: {Timestamp:2026-07-02 13:13:55.491952697 +0000 UTC m=+0.336666365 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099962 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23651355e812dcc17c572a7cbd82e5 SystemUUID:ec236513-55e8-12dc-c17c-572a7cbd82e5 BootID:a6812e5f-5acd-4281-ae7f-5c5759b06c8c Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1f:7e:5f:7e:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1f:7e:5f:7e:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:39:4d:65:4f:13 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 02 13:13:55.879770 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.494140 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 02 13:13:55.879770 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.494210 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 02 13:13:55.879770 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.497702 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 02 13:13:55.879770 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.497724 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-243.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 02 13:13:55.879770 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.498928 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 02 13:13:55.879770 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.498940 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 02 13:13:55.879770 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.498947 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.499686 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.500440 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.500517 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.504469 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.504573 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.504598 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.504607 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.504619 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.504627 2578 kubelet.go:398] "Adding apiserver pod source" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.504636 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.504929 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.504951 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.516093 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.524322 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.524353 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.525044 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.525062 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.525070 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.525076 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.525084 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.525090 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.525187 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.525199 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.525206 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.525211 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.525224 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.525232 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.526020 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.526028 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.528820 2578 server.go:1268] "Started kubelet" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.528874 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.528902 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.528998 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.529624 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fshtq" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.530732 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.531390 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.533296 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.538202 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.538724 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.539675 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-243.ec2.internal\" not found" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.540234 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.540248 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 02 13:13:55.880099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.540464 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.540530 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-243.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.540773 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.540804 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.540857 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.540965 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-243.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.541158 2578 factory.go:55] Registering systemd factory Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.541172 2578 factory.go:223] Registration of the systemd container factory successfully Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.541414 2578 factory.go:153] Registering CRI-O factory Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.541431 2578 factory.go:223] Registration of the crio container factory successfully Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.541487 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.541520 2578 factory.go:103] Registering Raw factory Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.541536 2578 manager.go:1196] Started watching for new ooms in manager Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.541809 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fshtq" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.541976 2578 manager.go:349] Starting recovery of all containers Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.541046 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-243.ec2.internal.18be7b443ee98ee1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-243.ec2.internal,UID:ip-10-0-131-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-243.ec2.internal,},FirstTimestamp:2026-07-02 13:13:55.528781537 +0000 UTC m=+0.373495206,LastTimestamp:2026-07-02 13:13:55.528781537 +0000 UTC m=+0.373495206,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-243.ec2.internal,}" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.553362 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.554164 2578 manager.go:354] Recovery completed Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.555748 2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.558929 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.559359 2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-243.ec2.internal\" not found" node="ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.560592 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasSufficientMemory" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.560619 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.560629 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasSufficientPID" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.561135 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.561147 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.561164 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.561166 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.561870 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasSufficientMemory" Jul 02 13:13:55.881333 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.561891 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.561902 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasSufficientPID" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.563540 2578 policy_none.go:49] "None policy: Start" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.563559 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.563574 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.565781 2578 policy_none.go:47] "Start" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.604652 2578 manager.go:337] "Starting Device Plugin manager" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.604691 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.604702 2578 server.go:87] "Starting device plugin registration server" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.604893 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.604903 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.606375 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.606447 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.606455 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.606984 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.607134 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-243.ec2.internal\" not found" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.617659 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.618875 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.618907 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.618926 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.618957 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.621434 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.707479 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.708304 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasSufficientMemory" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.708329 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.708342 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasSufficientPID" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.708370 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.715918 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.715935 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-243.ec2.internal\": node \"ip-10-0-131-243.ec2.internal\" not found" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.727890 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal"] Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.727957 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.730555 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasSufficientMemory" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.730590 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.730603 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasSufficientPID" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.732866 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.733005 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.733030 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.733624 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasSufficientMemory" Jul 02 13:13:55.882446 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.733659 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.733624 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasSufficientMemory" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.733695 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.733708 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasSufficientPID" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.733669 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasSufficientPID" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.735750 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.735771 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.736462 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasSufficientMemory" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.736490 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.736503 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeHasSufficientPID" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.742082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d935d27e6506ff6c126f79e137292824-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal\" (UID: \"d935d27e6506ff6c126f79e137292824\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.742110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d935d27e6506ff6c126f79e137292824-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal\" (UID: \"d935d27e6506ff6c126f79e137292824\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.742132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41c092df77332bb9c2b6523d6044f2bf-config\") pod \"kube-apiserver-proxy-ip-10-0-131-243.ec2.internal\" (UID: \"41c092df77332bb9c2b6523d6044f2bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.751554 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-243.ec2.internal\" not found" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.760551 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-243.ec2.internal\" not found" node="ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.764087 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-243.ec2.internal\" not found" node="ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.843169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d935d27e6506ff6c126f79e137292824-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal\" (UID: \"d935d27e6506ff6c126f79e137292824\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.843208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d935d27e6506ff6c126f79e137292824-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal\" (UID: \"d935d27e6506ff6c126f79e137292824\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.843211 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d935d27e6506ff6c126f79e137292824-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal\" (UID: \"d935d27e6506ff6c126f79e137292824\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.883537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.843226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41c092df77332bb9c2b6523d6044f2bf-config\") pod \"kube-apiserver-proxy-ip-10-0-131-243.ec2.internal\" (UID: \"41c092df77332bb9c2b6523d6044f2bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.884223 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.843261 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d935d27e6506ff6c126f79e137292824-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal\" (UID: \"d935d27e6506ff6c126f79e137292824\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.884223 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:55.843266 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41c092df77332bb9c2b6523d6044f2bf-config\") pod \"kube-apiserver-proxy-ip-10-0-131-243.ec2.internal\" (UID: \"41c092df77332bb9c2b6523d6044f2bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal" Jul 02 13:13:55.884223 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.852237 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-243.ec2.internal\" not found" Jul 02 13:13:56.071563 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:55.952639 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-243.ec2.internal\" not found" Jul 02 13:13:56.071563 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:56.053790 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-243.ec2.internal\" not found" Jul 02 13:13:56.071563 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.063627 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal" Jul 02 13:13:56.071563 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.066979 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal" Jul 02 13:13:56.198761 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:56.154354 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-243.ec2.internal\" not found" Jul 02 13:13:56.399696 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:56.254410 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-243.ec2.internal\" not found" Jul 02 13:13:56.399696 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:56.354712 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-243.ec2.internal\" not found" Jul 02 13:13:56.569690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.453455 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 13:13:56.569690 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:56.455681 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-243.ec2.internal\" not found" Jul 02 13:13:56.569690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.464117 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 02 13:13:56.569690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.464250 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 13:13:56.569690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.464279 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 13:13:56.569690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.464295 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 13:13:56.569690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.547478 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 13:08:55 +0000 UTC" deadline="2028-02-16 11:00:15.647037717 +0000 UTC" Jul 02 13:13:56.569690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.547504 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14253h46m19.099536661s" Jul 02 13:13:56.569690 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:56.556498 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-243.ec2.internal\" not found" Jul 02 13:13:56.681687 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:56.656761 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-243.ec2.internal\" not found" Jul 02 13:13:56.824254 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:56.757513 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-243.ec2.internal\" not found" Jul 02 13:13:56.824254 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:56.792965 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd935d27e6506ff6c126f79e137292824.slice/crio-64e69197dbfd04bccf11b20a7419c68198ecde05d9a7b7f5095c878c797d2c5b WatchSource:0}: Error finding container 64e69197dbfd04bccf11b20a7419c68198ecde05d9a7b7f5095c878c797d2c5b: Status 404 returned error can't find the container with id 64e69197dbfd04bccf11b20a7419c68198ecde05d9a7b7f5095c878c797d2c5b Jul 02 13:13:56.824254 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:56.793356 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c092df77332bb9c2b6523d6044f2bf.slice/crio-087728c92275802546e7312a88f8ac69c8b4ee6d8a26327b988a9c84b373c26e WatchSource:0}: Error finding container 087728c92275802546e7312a88f8ac69c8b4ee6d8a26327b988a9c84b373c26e: Status 404 returned error can't find the container with id 087728c92275802546e7312a88f8ac69c8b4ee6d8a26327b988a9c84b373c26e Jul 02 13:13:56.824254 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.800659 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 13:13:57.014343 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.847196 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 13:13:57.014343 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.945230 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal" Jul 02 13:13:57.014343 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.961731 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 13:13:57.014343 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.962966 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal" Jul 02 13:13:57.014343 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:56.974174 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 13:13:57.505503 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.505480 2578 apiserver.go:52] "Watching apiserver" Jul 02 13:13:57.514136 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.514110 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 02 13:13:57.517309 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.517105 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-j8m57","kube-system/konnectivity-agent-rqfvz","kube-system/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jv2bn","openshift-dns/node-resolver-rf9bz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal","openshift-multus/multus-4p9hq","openshift-multus/multus-additional-cni-plugins-zvkd2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp","openshift-image-registry/node-ca-c4k2p","openshift-multus/network-metrics-daemon-nmc96","openshift-network-diagnostics/network-check-target-g5qbq","openshift-network-operator/iptables-alerter-qj98s","openshift-ovn-kubernetes/ovnkube-node-zbwhh"] Jul 02 13:13:57.520069 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.520049 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.522553 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.522368 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qj98s" Jul 02 13:13:57.523234 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.523217 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 02 13:13:57.523480 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.523465 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 02 13:13:57.523665 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.523652 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-47nlq\"" Jul 02 13:13:57.523710 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.523661 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 02 13:13:57.523983 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.523965 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 02 13:13:57.526046 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.525314 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sb956\"" Jul 02 13:13:57.526046 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.525430 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 02 13:13:57.526046 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.525548 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 02 13:13:57.526706 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.526290 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.528858 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.528831 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.530895 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.529953 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 02 13:13:57.530895 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.530122 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 02 13:13:57.531455 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.531175 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rf9bz" Jul 02 13:13:57.532722 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.532706 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 02 13:13:57.533797 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.533782 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.533893 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.533877 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 02 13:13:57.534175 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.534154 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nfq5w\"" Jul 02 13:13:57.534479 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.534463 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 02 13:13:57.534479 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.534479 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 02 13:13:57.536417 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.536238 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.538675 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.538659 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:13:57.538773 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:57.538729 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:13:57.541611 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.540967 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 02 13:13:57.541611 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.541070 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rqfvz" Jul 02 13:13:57.542845 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.542526 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 02 13:13:57.542845 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.542743 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p6cw6\"" Jul 02 13:13:57.542845 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.542811 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 02 13:13:57.543176 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.542859 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 02 13:13:57.543176 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.543089 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 02 13:13:57.543176 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.543133 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 02 13:13:57.543176 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.543137 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 02 13:13:57.543397 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.542526 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 02 13:13:57.543451 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.543440 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z4w8t\"" Jul 02 13:13:57.543632 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.543615 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 02 13:13:57.543791 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.543773 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 02 13:13:57.543862 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.543789 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b86vc\"" Jul 02 13:13:57.544116 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.544097 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vvjkt\"" Jul 02 13:13:57.544231 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.544212 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 02 13:13:57.544388 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.544368 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 02 13:13:57.544701 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.544678 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7rf9g\"" Jul 02 13:13:57.545829 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.545669 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 02 13:13:57.546290 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.546271 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c4k2p" Jul 02 13:13:57.549170 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.549154 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:13:57.549436 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:57.549413 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:13:57.550680 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.550228 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 02 13:13:57.550680 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.550430 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-49kxc\"" Jul 02 13:13:57.550680 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.550470 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 02 13:13:57.550680 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.550237 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 02 13:13:57.551950 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.551844 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:13:57.551950 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:57.551915 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:13:57.560001 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.559798 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 13:13:57.579963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.578984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/305c0bc7-b833-4bca-8f4e-6f4a392f09b0-host\") pod \"node-ca-c4k2p\" (UID: \"305c0bc7-b833-4bca-8f4e-6f4a392f09b0\") " pod="openshift-image-registry/node-ca-c4k2p" Jul 02 13:13:57.579963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579020 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-multus-socket-dir-parent\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.579963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-socket-dir\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.579963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579060 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.579963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2787138-2777-487c-89da-f039c1f41ad2-os-release\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.579963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md49w\" (UniqueName: \"kubernetes.io/projected/305c0bc7-b833-4bca-8f4e-6f4a392f09b0-kube-api-access-md49w\") pod \"node-ca-c4k2p\" (UID: \"305c0bc7-b833-4bca-8f4e-6f4a392f09b0\") " pod="openshift-image-registry/node-ca-c4k2p" Jul 02 13:13:57.579963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-cnibin\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.579963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-multus-conf-dir\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.579963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-run-multus-certs\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.579963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5scmg\" (UniqueName: \"kubernetes.io/projected/f9481a84-1265-415b-8a3e-d2c8bfff0ada-kube-api-access-5scmg\") pod \"node-resolver-rf9bz\" (UID: \"f9481a84-1265-415b-8a3e-d2c8bfff0ada\") " pod="openshift-dns/node-resolver-rf9bz" Jul 02 13:13:57.579963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2787138-2777-487c-89da-f039c1f41ad2-cnibin\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.579963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579246 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-kubernetes\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.579963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-tmp\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.579963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn7nq\" (UniqueName: \"kubernetes.io/projected/9da8091c-ad4d-4134-9370-11e0e8075f0e-kube-api-access-xn7nq\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.581197 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579318 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-var-lib-cni-multus\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.581197 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-var-lib-kubelet\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.581197 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579374 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-etc-openvswitch\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.581197 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c443c208-241a-4759-a5d9-657a8ff485c2-kubelet-config\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:13:57.581197 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579432 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/305c0bc7-b833-4bca-8f4e-6f4a392f09b0-serviceca\") pod \"node-ca-c4k2p\" (UID: \"305c0bc7-b833-4bca-8f4e-6f4a392f09b0\") " pod="openshift-image-registry/node-ca-c4k2p" Jul 02 13:13:57.581197 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579459 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/493bc598-29fb-4757-b9a7-b8bb1b346b77-cni-binary-copy\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.581197 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-node-log\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.581197 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.581197 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2787138-2777-487c-89da-f039c1f41ad2-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.581197 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-sys\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.581197 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579589 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.581197 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579617 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-sys-fs\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.581197 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/493bc598-29fb-4757-b9a7-b8bb1b346b77-multus-daemon-config\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.581197 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67cn5\" (UniqueName: \"kubernetes.io/projected/493bc598-29fb-4757-b9a7-b8bb1b346b77-kube-api-access-67cn5\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.581878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-env-overrides\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.581878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579730 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9481a84-1265-415b-8a3e-d2c8bfff0ada-tmp-dir\") pod \"node-resolver-rf9bz\" (UID: \"f9481a84-1265-415b-8a3e-d2c8bfff0ada\") " pod="openshift-dns/node-resolver-rf9bz" Jul 02 13:13:57.581878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-run\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.581878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579785 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a98330d-24e4-490f-810a-bc2488b8e20e-agent-certs\") pod \"konnectivity-agent-rqfvz\" (UID: \"9a98330d-24e4-490f-810a-bc2488b8e20e\") " pod="kube-system/konnectivity-agent-rqfvz" Jul 02 13:13:57.581878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-kubelet\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.581878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-ovn-node-metrics-cert\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.581878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.579873 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-tuned\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.581878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.580758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a98330d-24e4-490f-810a-bc2488b8e20e-konnectivity-ca\") pod \"konnectivity-agent-rqfvz\" (UID: \"9a98330d-24e4-490f-810a-bc2488b8e20e\") " pod="kube-system/konnectivity-agent-rqfvz" Jul 02 13:13:57.581878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.580807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bfeff88e-7125-4cf9-a2da-0e30027f22f6-iptables-alerter-script\") pod \"iptables-alerter-qj98s\" (UID: \"bfeff88e-7125-4cf9-a2da-0e30027f22f6\") " pod="openshift-network-operator/iptables-alerter-qj98s" Jul 02 13:13:57.581878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.580843 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-var-lib-openvswitch\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.581878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.580883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2787138-2777-487c-89da-f039c1f41ad2-system-cni-dir\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.581878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.580916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2787138-2777-487c-89da-f039c1f41ad2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.581878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.580963 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c443c208-241a-4759-a5d9-657a8ff485c2-dbus\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:13:57.581878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-systemd\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.582586 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-system-cni-dir\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.582586 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581062 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-run-ovn\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.582586 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-cni-netd\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.582586 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4pnr\" (UniqueName: \"kubernetes.io/projected/bfeff88e-7125-4cf9-a2da-0e30027f22f6-kube-api-access-f4pnr\") pod \"iptables-alerter-qj98s\" (UID: \"bfeff88e-7125-4cf9-a2da-0e30027f22f6\") " pod="openshift-network-operator/iptables-alerter-qj98s" Jul 02 13:13:57.582586 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-run-systemd\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.582586 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581201 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-etc-kubernetes\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.582586 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-ovnkube-config\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.582586 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-ovnkube-script-lib\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.582586 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2787138-2777-487c-89da-f039c1f41ad2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.582586 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-os-release\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.582586 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-systemd-units\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.582586 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-run-netns\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.582586 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581398 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-log-socket\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.582586 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqcgv\" (UniqueName: \"kubernetes.io/projected/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-kube-api-access-rqcgv\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.583302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581448 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-modprobe-d\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.583302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581471 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-sysctl-d\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.583302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-host\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.583302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-multus-cni-dir\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.583302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfeff88e-7125-4cf9-a2da-0e30027f22f6-host-slash\") pod \"iptables-alerter-qj98s\" (UID: \"bfeff88e-7125-4cf9-a2da-0e30027f22f6\") " pod="openshift-network-operator/iptables-alerter-qj98s" Jul 02 13:13:57.583302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581561 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-etc-selinux\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.583302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-run-k8s-cni-cncf-io\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.583302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581612 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-run-netns\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.583302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-hostroot\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.583302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-registration-dir\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.583302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581727 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt8zq\" (UniqueName: \"kubernetes.io/projected/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-kube-api-access-pt8zq\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.583302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-slash\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.583302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-run-openvswitch\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.583302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9481a84-1265-415b-8a3e-d2c8bfff0ada-hosts-file\") pod \"node-resolver-rf9bz\" (UID: \"f9481a84-1265-415b-8a3e-d2c8bfff0ada\") " pod="openshift-dns/node-resolver-rf9bz" Jul 02 13:13:57.584177 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581834 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-sysctl-conf\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.584177 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581912 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-var-lib-kubelet\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.584177 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.581941 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-device-dir\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.584177 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.582009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-var-lib-cni-bin\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.584177 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.582045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-cni-bin\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.584177 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.582074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2787138-2777-487c-89da-f039c1f41ad2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.584177 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.582100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xf6z\" (UniqueName: \"kubernetes.io/projected/d2787138-2777-487c-89da-f039c1f41ad2-kube-api-access-6xf6z\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.584177 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.582123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:13:57.584177 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.582151 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-sysconfig\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.584177 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.582176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-lib-modules\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.596668 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.596644 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4224c" Jul 02 13:13:57.610286 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.609874 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4224c" Jul 02 13:13:57.627077 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.627026 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal" event={"ID":"41c092df77332bb9c2b6523d6044f2bf","Type":"ContainerStarted","Data":"087728c92275802546e7312a88f8ac69c8b4ee6d8a26327b988a9c84b373c26e"} Jul 02 13:13:57.628436 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.628411 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal" event={"ID":"d935d27e6506ff6c126f79e137292824","Type":"ContainerStarted","Data":"64e69197dbfd04bccf11b20a7419c68198ecde05d9a7b7f5095c878c797d2c5b"} Jul 02 13:13:57.641403 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.641372 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-run-k8s-cni-cncf-io\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-run-netns\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-hostroot\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-registration-dir\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt8zq\" (UniqueName: \"kubernetes.io/projected/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-kube-api-access-pt8zq\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-slash\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-run-openvswitch\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683471 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-run-netns\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9481a84-1265-415b-8a3e-d2c8bfff0ada-hosts-file\") pod \"node-resolver-rf9bz\" (UID: \"f9481a84-1265-415b-8a3e-d2c8bfff0ada\") " pod="openshift-dns/node-resolver-rf9bz" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683512 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-run-openvswitch\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-sysctl-conf\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683553 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-hostroot\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-var-lib-kubelet\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-device-dir\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.686879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683608 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-registration-dir\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.687762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683617 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-var-lib-cni-bin\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.687762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683664 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-var-lib-cni-bin\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.687762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683703 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-run-k8s-cni-cncf-io\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.687762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683749 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9481a84-1265-415b-8a3e-d2c8bfff0ada-hosts-file\") pod \"node-resolver-rf9bz\" (UID: \"f9481a84-1265-415b-8a3e-d2c8bfff0ada\") " pod="openshift-dns/node-resolver-rf9bz" Jul 02 13:13:57.687762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683862 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-sysctl-conf\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.687762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683872 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-slash\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.687762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683901 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-cni-bin\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.687762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2787138-2777-487c-89da-f039c1f41ad2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.687762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xf6z\" (UniqueName: \"kubernetes.io/projected/d2787138-2777-487c-89da-f039c1f41ad2-kube-api-access-6xf6z\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.687762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684031 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:13:57.687762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-sysconfig\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.687762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-lib-modules\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.687762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684112 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/305c0bc7-b833-4bca-8f4e-6f4a392f09b0-host\") pod \"node-ca-c4k2p\" (UID: \"305c0bc7-b833-4bca-8f4e-6f4a392f09b0\") " pod="openshift-image-registry/node-ca-c4k2p" Jul 02 13:13:57.687762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-multus-socket-dir-parent\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.688463 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-socket-dir\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.688463 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684187 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.688463 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2787138-2777-487c-89da-f039c1f41ad2-os-release\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.688463 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md49w\" (UniqueName: \"kubernetes.io/projected/305c0bc7-b833-4bca-8f4e-6f4a392f09b0-kube-api-access-md49w\") pod \"node-ca-c4k2p\" (UID: \"305c0bc7-b833-4bca-8f4e-6f4a392f09b0\") " pod="openshift-image-registry/node-ca-c4k2p" Jul 02 13:13:57.688463 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-cnibin\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.688463 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-multus-conf-dir\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.688463 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684333 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-run-multus-certs\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.688463 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684362 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs\") pod \"network-metrics-daemon-nmc96\" (UID: \"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4\") " pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:13:57.688463 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5scmg\" (UniqueName: \"kubernetes.io/projected/f9481a84-1265-415b-8a3e-d2c8bfff0ada-kube-api-access-5scmg\") pod \"node-resolver-rf9bz\" (UID: \"f9481a84-1265-415b-8a3e-d2c8bfff0ada\") " pod="openshift-dns/node-resolver-rf9bz" Jul 02 13:13:57.688463 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684397 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-var-lib-kubelet\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.688463 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2787138-2777-487c-89da-f039c1f41ad2-cnibin\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.688463 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684455 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-cni-bin\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.688463 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-kubernetes\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.688463 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-tmp\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.689172 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7nq\" (UniqueName: \"kubernetes.io/projected/9da8091c-ad4d-4134-9370-11e0e8075f0e-kube-api-access-xn7nq\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.689172 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-var-lib-cni-multus\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.689172 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:57.684554 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 13:13:57.689172 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-var-lib-kubelet\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.689172 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-etc-openvswitch\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.689172 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:57.684637 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret podName:c443c208-241a-4759-a5d9-657a8ff485c2 nodeName:}" failed. No retries permitted until 2026-07-02 13:13:58.184603054 +0000 UTC m=+3.029316727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret") pod "global-pull-secret-syncer-j8m57" (UID: "c443c208-241a-4759-a5d9-657a8ff485c2") : object "kube-system"/"original-pull-secret" not registered Jul 02 13:13:57.689172 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684664 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c443c208-241a-4759-a5d9-657a8ff485c2-kubelet-config\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:13:57.689172 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684693 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/305c0bc7-b833-4bca-8f4e-6f4a392f09b0-serviceca\") pod \"node-ca-c4k2p\" (UID: \"305c0bc7-b833-4bca-8f4e-6f4a392f09b0\") " pod="openshift-image-registry/node-ca-c4k2p" Jul 02 13:13:57.689172 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/493bc598-29fb-4757-b9a7-b8bb1b346b77-cni-binary-copy\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.689172 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-node-log\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.689172 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.689172 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684811 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2787138-2777-487c-89da-f039c1f41ad2-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.689172 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684845 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-sys\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.689172 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.689872 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-sys-fs\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.689872 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/493bc598-29fb-4757-b9a7-b8bb1b346b77-multus-daemon-config\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.689872 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67cn5\" (UniqueName: \"kubernetes.io/projected/493bc598-29fb-4757-b9a7-b8bb1b346b77-kube-api-access-67cn5\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.689872 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685011 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-env-overrides\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.689872 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9481a84-1265-415b-8a3e-d2c8bfff0ada-tmp-dir\") pod \"node-resolver-rf9bz\" (UID: \"f9481a84-1265-415b-8a3e-d2c8bfff0ada\") " pod="openshift-dns/node-resolver-rf9bz" Jul 02 13:13:57.689872 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-run\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.689872 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a98330d-24e4-490f-810a-bc2488b8e20e-agent-certs\") pod \"konnectivity-agent-rqfvz\" (UID: \"9a98330d-24e4-490f-810a-bc2488b8e20e\") " pod="kube-system/konnectivity-agent-rqfvz" Jul 02 13:13:57.689872 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685117 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-kubelet\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.689872 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.684635 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-etc-openvswitch\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.689872 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-ovn-node-metrics-cert\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.689872 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.683928 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-device-dir\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.689872 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685209 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-tuned\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.689872 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685229 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-kubernetes\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.689872 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685239 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-sys-fs\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685249 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2787138-2777-487c-89da-f039c1f41ad2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a98330d-24e4-490f-810a-bc2488b8e20e-konnectivity-ca\") pod \"konnectivity-agent-rqfvz\" (UID: \"9a98330d-24e4-490f-810a-bc2488b8e20e\") " pod="kube-system/konnectivity-agent-rqfvz" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685243 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c443c208-241a-4759-a5d9-657a8ff485c2-kubelet-config\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685509 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-kubelet\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685511 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bfeff88e-7125-4cf9-a2da-0e30027f22f6-iptables-alerter-script\") pod \"iptables-alerter-qj98s\" (UID: \"bfeff88e-7125-4cf9-a2da-0e30027f22f6\") " pod="openshift-network-operator/iptables-alerter-qj98s" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685568 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-run-multus-certs\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685634 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685700 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2787138-2777-487c-89da-f039c1f41ad2-os-release\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685813 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/493bc598-29fb-4757-b9a7-b8bb1b346b77-cni-binary-copy\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.685947 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2787138-2777-487c-89da-f039c1f41ad2-cnibin\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686015 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/305c0bc7-b833-4bca-8f4e-6f4a392f09b0-serviceca\") pod \"node-ca-c4k2p\" (UID: \"305c0bc7-b833-4bca-8f4e-6f4a392f09b0\") " pod="openshift-image-registry/node-ca-c4k2p" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686030 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-sys\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-var-lib-openvswitch\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.690607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2787138-2777-487c-89da-f039c1f41ad2-system-cni-dir\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.691298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686115 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2787138-2777-487c-89da-f039c1f41ad2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.691298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c443c208-241a-4759-a5d9-657a8ff485c2-dbus\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:13:57.691298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-systemd\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.691298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-system-cni-dir\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.691298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-run-ovn\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.691298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686276 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.691298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-cni-netd\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.691298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686316 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-node-log\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.691298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frctg\" (UniqueName: \"kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg\") pod \"network-check-target-g5qbq\" (UID: \"eccf7e61-76e3-4853-aa14-be343c592a5d\") " pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:13:57.691298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686346 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-var-lib-cni-multus\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.691298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686351 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2787138-2777-487c-89da-f039c1f41ad2-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.691298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686355 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4pnr\" (UniqueName: \"kubernetes.io/projected/bfeff88e-7125-4cf9-a2da-0e30027f22f6-kube-api-access-f4pnr\") pod \"iptables-alerter-qj98s\" (UID: \"bfeff88e-7125-4cf9-a2da-0e30027f22f6\") " pod="openshift-network-operator/iptables-alerter-qj98s" Jul 02 13:13:57.691298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686403 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-run-systemd\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.691298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686432 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-etc-kubernetes\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686453 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9481a84-1265-415b-8a3e-d2c8bfff0ada-tmp-dir\") pod \"node-resolver-rf9bz\" (UID: \"f9481a84-1265-415b-8a3e-d2c8bfff0ada\") " pod="openshift-dns/node-resolver-rf9bz" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-ovnkube-config\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686492 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-host-var-lib-kubelet\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686516 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-ovnkube-script-lib\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686539 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-cnibin\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2787138-2777-487c-89da-f039c1f41ad2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-os-release\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686584 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-sysconfig\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg6pz\" (UniqueName: \"kubernetes.io/projected/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-kube-api-access-mg6pz\") pod \"network-metrics-daemon-nmc96\" (UID: \"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4\") " pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-systemd-units\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686659 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-run-netns\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-log-socket\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686704 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-run\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqcgv\" (UniqueName: \"kubernetes.io/projected/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-kube-api-access-rqcgv\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.691969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-modprobe-d\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686743 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-multus-conf-dir\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686772 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/305c0bc7-b833-4bca-8f4e-6f4a392f09b0-host\") pod \"node-ca-c4k2p\" (UID: \"305c0bc7-b833-4bca-8f4e-6f4a392f09b0\") " pod="openshift-image-registry/node-ca-c4k2p" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686849 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/493bc598-29fb-4757-b9a7-b8bb1b346b77-multus-daemon-config\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686870 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-systemd-units\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686877 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-run-netns\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-sysctl-d\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686932 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-multus-socket-dir-parent\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686950 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-etc-kubernetes\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686971 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-modprobe-d\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687036 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687085 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-env-overrides\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.686164 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bfeff88e-7125-4cf9-a2da-0e30027f22f6-iptables-alerter-script\") pod \"iptables-alerter-qj98s\" (UID: \"bfeff88e-7125-4cf9-a2da-0e30027f22f6\") " pod="openshift-network-operator/iptables-alerter-qj98s" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687285 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c443c208-241a-4759-a5d9-657a8ff485c2-dbus\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687350 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-systemd\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.692798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687403 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-system-cni-dir\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687447 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2787138-2777-487c-89da-f039c1f41ad2-system-cni-dir\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687505 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2787138-2777-487c-89da-f039c1f41ad2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687551 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-socket-dir\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-ovnkube-config\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687571 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-os-release\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-var-lib-openvswitch\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687827 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2787138-2777-487c-89da-f039c1f41ad2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687904 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-sysctl-d\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687911 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-log-socket\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687947 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-host-cni-netd\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-host\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.687982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-multus-cni-dir\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.688024 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-run-ovn\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.688031 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfeff88e-7125-4cf9-a2da-0e30027f22f6-host-slash\") pod \"iptables-alerter-qj98s\" (UID: \"bfeff88e-7125-4cf9-a2da-0e30027f22f6\") " pod="openshift-network-operator/iptables-alerter-qj98s" Jul 02 13:13:57.693569 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.688091 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/493bc598-29fb-4757-b9a7-b8bb1b346b77-multus-cni-dir\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.694371 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.688112 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-ovnkube-script-lib\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.694371 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.688109 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfeff88e-7125-4cf9-a2da-0e30027f22f6-host-slash\") pod \"iptables-alerter-qj98s\" (UID: \"bfeff88e-7125-4cf9-a2da-0e30027f22f6\") " pod="openshift-network-operator/iptables-alerter-qj98s" Jul 02 13:13:57.694371 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.688151 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a98330d-24e4-490f-810a-bc2488b8e20e-konnectivity-ca\") pod \"konnectivity-agent-rqfvz\" (UID: \"9a98330d-24e4-490f-810a-bc2488b8e20e\") " pod="kube-system/konnectivity-agent-rqfvz" Jul 02 13:13:57.694371 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.688177 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-etc-selinux\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.694371 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.688213 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-host\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.694371 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.688223 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-lib-modules\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.694371 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.688280 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-run-systemd\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.694371 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.688307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9da8091c-ad4d-4134-9370-11e0e8075f0e-etc-selinux\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.694371 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.692122 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-tmp\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.694371 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.692405 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-etc-tuned\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.695542 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.695519 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-ovn-node-metrics-cert\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.695820 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.695801 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 13:13:57.698277 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.697422 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt8zq\" (UniqueName: \"kubernetes.io/projected/4e35f04b-fa9b-4dcf-80fe-d198d261ae2e-kube-api-access-pt8zq\") pod \"tuned-jv2bn\" (UID: \"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e\") " pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.698277 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.697822 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a98330d-24e4-490f-810a-bc2488b8e20e-agent-certs\") pod \"konnectivity-agent-rqfvz\" (UID: \"9a98330d-24e4-490f-810a-bc2488b8e20e\") " pod="kube-system/konnectivity-agent-rqfvz" Jul 02 13:13:57.698277 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.698235 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xf6z\" (UniqueName: \"kubernetes.io/projected/d2787138-2777-487c-89da-f039c1f41ad2-kube-api-access-6xf6z\") pod \"multus-additional-cni-plugins-zvkd2\" (UID: \"d2787138-2777-487c-89da-f039c1f41ad2\") " pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.700864 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.700812 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5scmg\" (UniqueName: \"kubernetes.io/projected/f9481a84-1265-415b-8a3e-d2c8bfff0ada-kube-api-access-5scmg\") pod \"node-resolver-rf9bz\" (UID: \"f9481a84-1265-415b-8a3e-d2c8bfff0ada\") " pod="openshift-dns/node-resolver-rf9bz" Jul 02 13:13:57.702373 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.702340 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4pnr\" (UniqueName: \"kubernetes.io/projected/bfeff88e-7125-4cf9-a2da-0e30027f22f6-kube-api-access-f4pnr\") pod \"iptables-alerter-qj98s\" (UID: \"bfeff88e-7125-4cf9-a2da-0e30027f22f6\") " pod="openshift-network-operator/iptables-alerter-qj98s" Jul 02 13:13:57.703050 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.702866 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqcgv\" (UniqueName: \"kubernetes.io/projected/3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11-kube-api-access-rqcgv\") pod \"ovnkube-node-zbwhh\" (UID: \"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.703443 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.703404 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md49w\" (UniqueName: \"kubernetes.io/projected/305c0bc7-b833-4bca-8f4e-6f4a392f09b0-kube-api-access-md49w\") pod \"node-ca-c4k2p\" (UID: \"305c0bc7-b833-4bca-8f4e-6f4a392f09b0\") " pod="openshift-image-registry/node-ca-c4k2p" Jul 02 13:13:57.704982 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.704959 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7nq\" (UniqueName: \"kubernetes.io/projected/9da8091c-ad4d-4134-9370-11e0e8075f0e-kube-api-access-xn7nq\") pod \"aws-ebs-csi-driver-node-lftgp\" (UID: \"9da8091c-ad4d-4134-9370-11e0e8075f0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.706423 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.706401 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67cn5\" (UniqueName: \"kubernetes.io/projected/493bc598-29fb-4757-b9a7-b8bb1b346b77-kube-api-access-67cn5\") pod \"multus-4p9hq\" (UID: \"493bc598-29fb-4757-b9a7-b8bb1b346b77\") " pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.789692 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.789143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs\") pod \"network-metrics-daemon-nmc96\" (UID: \"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4\") " pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:13:57.789692 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.789203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frctg\" (UniqueName: \"kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg\") pod \"network-check-target-g5qbq\" (UID: \"eccf7e61-76e3-4853-aa14-be343c592a5d\") " pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:13:57.789692 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.789235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg6pz\" (UniqueName: \"kubernetes.io/projected/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-kube-api-access-mg6pz\") pod \"network-metrics-daemon-nmc96\" (UID: \"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4\") " pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:13:57.789692 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:57.789541 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 13:13:57.789692 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:57.789604 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs podName:ac5d1704-1da6-48e2-b5fa-edecec3a5ee4 nodeName:}" failed. No retries permitted until 2026-07-02 13:13:58.289584905 +0000 UTC m=+3.134298566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs") pod "network-metrics-daemon-nmc96" (UID: "ac5d1704-1da6-48e2-b5fa-edecec3a5ee4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 13:13:57.804126 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.804103 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 13:13:57.810041 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:57.809777 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 13:13:57.810041 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:57.809806 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 13:13:57.810041 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:57.809821 2578 projected.go:196] Error preparing data for projected volume kube-api-access-frctg for pod openshift-network-diagnostics/network-check-target-g5qbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 13:13:57.810041 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:57.809878 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg podName:eccf7e61-76e3-4853-aa14-be343c592a5d nodeName:}" failed. No retries permitted until 2026-07-02 13:13:58.309860994 +0000 UTC m=+3.154574659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-frctg" (UniqueName: "kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg") pod "network-check-target-g5qbq" (UID: "eccf7e61-76e3-4853-aa14-be343c592a5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 13:13:57.814389 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.814365 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg6pz\" (UniqueName: \"kubernetes.io/projected/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-kube-api-access-mg6pz\") pod \"network-metrics-daemon-nmc96\" (UID: \"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4\") " pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:13:57.834183 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.834153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4p9hq" Jul 02 13:13:57.850004 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.849969 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qj98s" Jul 02 13:13:57.851462 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:57.851437 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493bc598_29fb_4757_b9a7_b8bb1b346b77.slice/crio-588eaae42c1ca362898b06a4f94e7558e7be4567ecf20470daf242051f962449 WatchSource:0}: Error finding container 588eaae42c1ca362898b06a4f94e7558e7be4567ecf20470daf242051f962449: Status 404 returned error can't find the container with id 588eaae42c1ca362898b06a4f94e7558e7be4567ecf20470daf242051f962449 Jul 02 13:13:57.860837 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.860815 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:13:57.861443 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:57.861420 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfeff88e_7125_4cf9_a2da_0e30027f22f6.slice/crio-27f8058139a5a2626a6861601d13a635469ce968d4e4967d7831399325b38404 WatchSource:0}: Error finding container 27f8058139a5a2626a6861601d13a635469ce968d4e4967d7831399325b38404: Status 404 returned error can't find the container with id 27f8058139a5a2626a6861601d13a635469ce968d4e4967d7831399325b38404 Jul 02 13:13:57.872326 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:57.872304 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffbc11c_dcdc_4ff7_b33f_01cd92de5f11.slice/crio-748db20d2e697618d2c9323869475f5cb492d938cfced46d62275bb86f9896ab WatchSource:0}: Error finding container 748db20d2e697618d2c9323869475f5cb492d938cfced46d62275bb86f9896ab: Status 404 returned error can't find the container with id 748db20d2e697618d2c9323869475f5cb492d938cfced46d62275bb86f9896ab Jul 02 13:13:57.877874 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.877577 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" Jul 02 13:13:57.887291 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.887270 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rf9bz" Jul 02 13:13:57.888000 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:57.887947 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e35f04b_fa9b_4dcf_80fe_d198d261ae2e.slice/crio-68476cbc69b56d436f2a2e0619655bea1720a14ca07f1ada3531da3fab68db84 WatchSource:0}: Error finding container 68476cbc69b56d436f2a2e0619655bea1720a14ca07f1ada3531da3fab68db84: Status 404 returned error can't find the container with id 68476cbc69b56d436f2a2e0619655bea1720a14ca07f1ada3531da3fab68db84 Jul 02 13:13:57.898820 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:57.898795 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9481a84_1265_415b_8a3e_d2c8bfff0ada.slice/crio-67ce0a88af7375b1b7039213634f79cc199ae9cc06a96f9c568f239d38819919 WatchSource:0}: Error finding container 67ce0a88af7375b1b7039213634f79cc199ae9cc06a96f9c568f239d38819919: Status 404 returned error can't find the container with id 67ce0a88af7375b1b7039213634f79cc199ae9cc06a96f9c568f239d38819919 Jul 02 13:13:57.902547 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.902528 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvkd2" Jul 02 13:13:57.911762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.911743 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" Jul 02 13:13:57.914192 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:57.914170 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2787138_2777_487c_89da_f039c1f41ad2.slice/crio-3c114da54077a5f0508454c0284e1714936de51cdb0297453908b6d3cc20b593 WatchSource:0}: Error finding container 3c114da54077a5f0508454c0284e1714936de51cdb0297453908b6d3cc20b593: Status 404 returned error can't find the container with id 3c114da54077a5f0508454c0284e1714936de51cdb0297453908b6d3cc20b593 Jul 02 13:13:57.920842 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.920666 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rqfvz" Jul 02 13:13:57.926617 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:57.926593 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da8091c_ad4d_4134_9370_11e0e8075f0e.slice/crio-d73528c2ddd546bf35603f41d3fffe9ad36b23ab72714c6a50cb7dc8c2b627b0 WatchSource:0}: Error finding container d73528c2ddd546bf35603f41d3fffe9ad36b23ab72714c6a50cb7dc8c2b627b0: Status 404 returned error can't find the container with id d73528c2ddd546bf35603f41d3fffe9ad36b23ab72714c6a50cb7dc8c2b627b0 Jul 02 13:13:57.926965 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:57.926789 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c4k2p" Jul 02 13:13:57.937784 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:57.937754 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a98330d_24e4_490f_810a_bc2488b8e20e.slice/crio-517808e95f9d0e92842ed1e2058544db70888673ff4879f80db00be46add273f WatchSource:0}: Error finding container 517808e95f9d0e92842ed1e2058544db70888673ff4879f80db00be46add273f: Status 404 returned error can't find the container with id 517808e95f9d0e92842ed1e2058544db70888673ff4879f80db00be46add273f Jul 02 13:13:57.940449 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:13:57.940426 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305c0bc7_b833_4bca_8f4e_6f4a392f09b0.slice/crio-973ceb832acb7d39de8ce71f6f7aa7f4541fda55552c6152a03898a365326b0d WatchSource:0}: Error finding container 973ceb832acb7d39de8ce71f6f7aa7f4541fda55552c6152a03898a365326b0d: Status 404 returned error can't find the container with id 973ceb832acb7d39de8ce71f6f7aa7f4541fda55552c6152a03898a365326b0d Jul 02 13:13:58.191712 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:58.191609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:13:58.191863 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:58.191793 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 13:13:58.191921 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:58.191868 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret podName:c443c208-241a-4759-a5d9-657a8ff485c2 nodeName:}" failed. No retries permitted until 2026-07-02 13:13:59.1918467 +0000 UTC m=+4.036560366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret") pod "global-pull-secret-syncer-j8m57" (UID: "c443c208-241a-4759-a5d9-657a8ff485c2") : object "kube-system"/"original-pull-secret" not registered Jul 02 13:13:58.293049 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:58.293016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs\") pod \"network-metrics-daemon-nmc96\" (UID: \"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4\") " pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:13:58.293249 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:58.293210 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 13:13:58.293311 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:58.293289 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs podName:ac5d1704-1da6-48e2-b5fa-edecec3a5ee4 nodeName:}" failed. No retries permitted until 2026-07-02 13:13:59.2932666 +0000 UTC m=+4.137980279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs") pod "network-metrics-daemon-nmc96" (UID: "ac5d1704-1da6-48e2-b5fa-edecec3a5ee4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 13:13:58.394387 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:58.394353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frctg\" (UniqueName: \"kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg\") pod \"network-check-target-g5qbq\" (UID: \"eccf7e61-76e3-4853-aa14-be343c592a5d\") " pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:13:58.394541 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:58.394502 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 13:13:58.394602 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:58.394523 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 13:13:58.394602 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:58.394567 2578 projected.go:196] Error preparing data for projected volume kube-api-access-frctg for pod openshift-network-diagnostics/network-check-target-g5qbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 13:13:58.394683 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:58.394635 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg podName:eccf7e61-76e3-4853-aa14-be343c592a5d nodeName:}" failed. No retries permitted until 2026-07-02 13:13:59.394611772 +0000 UTC m=+4.239325452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-frctg" (UniqueName: "kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg") pod "network-check-target-g5qbq" (UID: "eccf7e61-76e3-4853-aa14-be343c592a5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 13:13:58.613377 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:58.613263 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:08:57 +0000 UTC" deadline="2028-04-05 17:30:21.56143938 +0000 UTC" Jul 02 13:13:58.613377 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:58.613303 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15436h16m22.948141377s" Jul 02 13:13:58.632581 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:58.632548 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c4k2p" event={"ID":"305c0bc7-b833-4bca-8f4e-6f4a392f09b0","Type":"ContainerStarted","Data":"973ceb832acb7d39de8ce71f6f7aa7f4541fda55552c6152a03898a365326b0d"} Jul 02 13:13:58.634847 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:58.634826 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rqfvz" event={"ID":"9a98330d-24e4-490f-810a-bc2488b8e20e","Type":"ContainerStarted","Data":"517808e95f9d0e92842ed1e2058544db70888673ff4879f80db00be46add273f"} Jul 02 13:13:58.636960 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:58.636860 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkd2" event={"ID":"d2787138-2777-487c-89da-f039c1f41ad2","Type":"ContainerStarted","Data":"3c114da54077a5f0508454c0284e1714936de51cdb0297453908b6d3cc20b593"} Jul 02 13:13:58.638380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:58.638314 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rf9bz" event={"ID":"f9481a84-1265-415b-8a3e-d2c8bfff0ada","Type":"ContainerStarted","Data":"67ce0a88af7375b1b7039213634f79cc199ae9cc06a96f9c568f239d38819919"} Jul 02 13:13:58.641103 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:58.641076 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" event={"ID":"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11","Type":"ContainerStarted","Data":"748db20d2e697618d2c9323869475f5cb492d938cfced46d62275bb86f9896ab"} Jul 02 13:13:58.645628 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:58.645599 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4p9hq" event={"ID":"493bc598-29fb-4757-b9a7-b8bb1b346b77","Type":"ContainerStarted","Data":"588eaae42c1ca362898b06a4f94e7558e7be4567ecf20470daf242051f962449"} Jul 02 13:13:58.648768 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:58.648739 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" event={"ID":"9da8091c-ad4d-4134-9370-11e0e8075f0e","Type":"ContainerStarted","Data":"d73528c2ddd546bf35603f41d3fffe9ad36b23ab72714c6a50cb7dc8c2b627b0"} Jul 02 13:13:58.650893 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:58.650864 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" event={"ID":"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e","Type":"ContainerStarted","Data":"68476cbc69b56d436f2a2e0619655bea1720a14ca07f1ada3531da3fab68db84"} Jul 02 13:13:58.655696 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:58.655645 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qj98s" event={"ID":"bfeff88e-7125-4cf9-a2da-0e30027f22f6","Type":"ContainerStarted","Data":"27f8058139a5a2626a6861601d13a635469ce968d4e4967d7831399325b38404"} Jul 02 13:13:59.203052 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:59.202445 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:13:59.203052 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:59.202614 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 13:13:59.203052 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:59.202680 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret podName:c443c208-241a-4759-a5d9-657a8ff485c2 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:01.202660081 +0000 UTC m=+6.047373761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret") pod "global-pull-secret-syncer-j8m57" (UID: "c443c208-241a-4759-a5d9-657a8ff485c2") : object "kube-system"/"original-pull-secret" not registered Jul 02 13:13:59.303774 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:59.303156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs\") pod \"network-metrics-daemon-nmc96\" (UID: \"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4\") " pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:13:59.303774 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:59.303338 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 13:13:59.303774 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:59.303403 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs podName:ac5d1704-1da6-48e2-b5fa-edecec3a5ee4 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:01.303383338 +0000 UTC m=+6.148096999 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs") pod "network-metrics-daemon-nmc96" (UID: "ac5d1704-1da6-48e2-b5fa-edecec3a5ee4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 13:13:59.404537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:59.403807 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frctg\" (UniqueName: \"kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg\") pod \"network-check-target-g5qbq\" (UID: \"eccf7e61-76e3-4853-aa14-be343c592a5d\") " pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:13:59.404537 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:59.404072 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 13:13:59.404537 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:59.404093 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 13:13:59.404537 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:59.404107 2578 projected.go:196] Error preparing data for projected volume kube-api-access-frctg for pod openshift-network-diagnostics/network-check-target-g5qbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 13:13:59.404537 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:59.404170 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg podName:eccf7e61-76e3-4853-aa14-be343c592a5d nodeName:}" failed. No retries permitted until 2026-07-02 13:14:01.404149405 +0000 UTC m=+6.248863071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-frctg" (UniqueName: "kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg") pod "network-check-target-g5qbq" (UID: "eccf7e61-76e3-4853-aa14-be343c592a5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 13:13:59.425154 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:59.424827 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 13:13:59.613774 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:59.613621 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:08:57 +0000 UTC" deadline="2028-03-18 10:29:19.307557234 +0000 UTC" Jul 02 13:13:59.613774 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:59.613686 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14997h15m19.693876027s" Jul 02 13:13:59.622516 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:59.619589 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:13:59.622516 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:59.619736 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:13:59.622516 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:59.620180 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:13:59.622516 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:59.620281 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:13:59.622516 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:13:59.620344 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:13:59.622516 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:13:59.620412 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:01.220418 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:01.219810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:01.220418 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:01.220002 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 13:14:01.220418 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:01.220066 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret podName:c443c208-241a-4759-a5d9-657a8ff485c2 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:05.220048605 +0000 UTC m=+10.064762264 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret") pod "global-pull-secret-syncer-j8m57" (UID: "c443c208-241a-4759-a5d9-657a8ff485c2") : object "kube-system"/"original-pull-secret" not registered Jul 02 13:14:01.321075 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:01.320604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs\") pod \"network-metrics-daemon-nmc96\" (UID: \"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4\") " pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:01.321075 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:01.320797 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 13:14:01.321075 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:01.320859 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs podName:ac5d1704-1da6-48e2-b5fa-edecec3a5ee4 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:05.320840993 +0000 UTC m=+10.165554652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs") pod "network-metrics-daemon-nmc96" (UID: "ac5d1704-1da6-48e2-b5fa-edecec3a5ee4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 13:14:01.421228 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:01.421194 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frctg\" (UniqueName: \"kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg\") pod \"network-check-target-g5qbq\" (UID: \"eccf7e61-76e3-4853-aa14-be343c592a5d\") " pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:01.421421 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:01.421358 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 13:14:01.421421 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:01.421378 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 13:14:01.421421 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:01.421392 2578 projected.go:196] Error preparing data for projected volume kube-api-access-frctg for pod openshift-network-diagnostics/network-check-target-g5qbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 13:14:01.421578 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:01.421452 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg podName:eccf7e61-76e3-4853-aa14-be343c592a5d nodeName:}" failed. No retries permitted until 2026-07-02 13:14:05.421433702 +0000 UTC m=+10.266147361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-frctg" (UniqueName: "kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg") pod "network-check-target-g5qbq" (UID: "eccf7e61-76e3-4853-aa14-be343c592a5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 13:14:01.619853 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:01.619770 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:01.620018 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:01.619907 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:01.622318 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:01.620283 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:01.622318 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:01.620375 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:01.622318 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:01.620414 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:01.622318 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:01.620463 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:03.619857 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:03.619769 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:03.620309 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:03.619892 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:03.620309 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:03.619769 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:03.620309 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:03.619967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:03.620309 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:03.620078 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:03.620309 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:03.620165 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:05.252008 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:05.251907 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:05.252381 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:05.252021 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 13:14:05.252381 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:05.252089 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret podName:c443c208-241a-4759-a5d9-657a8ff485c2 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:13.252068823 +0000 UTC m=+18.096782482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret") pod "global-pull-secret-syncer-j8m57" (UID: "c443c208-241a-4759-a5d9-657a8ff485c2") : object "kube-system"/"original-pull-secret" not registered Jul 02 13:14:05.352770 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:05.352738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs\") pod \"network-metrics-daemon-nmc96\" (UID: \"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4\") " pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:05.352972 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:05.352866 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 13:14:05.352972 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:05.352948 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs podName:ac5d1704-1da6-48e2-b5fa-edecec3a5ee4 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:13.352911929 +0000 UTC m=+18.197625588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs") pod "network-metrics-daemon-nmc96" (UID: "ac5d1704-1da6-48e2-b5fa-edecec3a5ee4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 13:14:05.453279 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:05.453247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frctg\" (UniqueName: \"kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg\") pod \"network-check-target-g5qbq\" (UID: \"eccf7e61-76e3-4853-aa14-be343c592a5d\") " pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:05.453445 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:05.453393 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 13:14:05.453445 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:05.453417 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 13:14:05.453445 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:05.453431 2578 projected.go:196] Error preparing data for projected volume kube-api-access-frctg for pod openshift-network-diagnostics/network-check-target-g5qbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 13:14:05.453632 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:05.453488 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg podName:eccf7e61-76e3-4853-aa14-be343c592a5d nodeName:}" failed. No retries permitted until 2026-07-02 13:14:13.453474389 +0000 UTC m=+18.298188044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-frctg" (UniqueName: "kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg") pod "network-check-target-g5qbq" (UID: "eccf7e61-76e3-4853-aa14-be343c592a5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 13:14:05.623108 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:05.623029 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:05.623269 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:05.623205 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:05.623325 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:05.623258 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:05.623325 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:05.623147 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:05.623452 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:05.623033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:05.623564 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:05.623547 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:07.622940 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:07.622793 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:07.622940 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:07.622910 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:07.623355 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:07.622947 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:07.623355 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:07.622905 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:07.623355 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:07.623026 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:07.623355 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:07.623109 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:09.622685 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:09.622323 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:09.622685 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:09.622448 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:09.622685 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:09.622473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:09.622685 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:09.622544 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:09.622685 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:09.622595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:09.622685 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:09.622651 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:11.619640 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:11.619604 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:11.620144 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:11.619735 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:11.620204 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:11.620166 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:11.620269 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:11.620248 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:11.620440 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:11.620407 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:11.620559 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:11.620530 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:13.312280 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:13.312244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:13.312781 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:13.312387 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 13:14:13.312781 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:13.312445 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret podName:c443c208-241a-4759-a5d9-657a8ff485c2 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:29.312428994 +0000 UTC m=+34.157142652 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret") pod "global-pull-secret-syncer-j8m57" (UID: "c443c208-241a-4759-a5d9-657a8ff485c2") : object "kube-system"/"original-pull-secret" not registered Jul 02 13:14:13.413162 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:13.413126 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs\") pod \"network-metrics-daemon-nmc96\" (UID: \"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4\") " pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:13.413352 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:13.413305 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 13:14:13.413401 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:13.413387 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs podName:ac5d1704-1da6-48e2-b5fa-edecec3a5ee4 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:29.413365488 +0000 UTC m=+34.258079149 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs") pod "network-metrics-daemon-nmc96" (UID: "ac5d1704-1da6-48e2-b5fa-edecec3a5ee4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 13:14:13.514397 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:13.514356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frctg\" (UniqueName: \"kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg\") pod \"network-check-target-g5qbq\" (UID: \"eccf7e61-76e3-4853-aa14-be343c592a5d\") " pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:13.514571 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:13.514511 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 13:14:13.514571 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:13.514530 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 13:14:13.514571 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:13.514544 2578 projected.go:196] Error preparing data for projected volume kube-api-access-frctg for pod openshift-network-diagnostics/network-check-target-g5qbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 13:14:13.514707 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:13.514613 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg podName:eccf7e61-76e3-4853-aa14-be343c592a5d nodeName:}" failed. No retries permitted until 2026-07-02 13:14:29.514599168 +0000 UTC m=+34.359312824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-frctg" (UniqueName: "kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg") pod "network-check-target-g5qbq" (UID: "eccf7e61-76e3-4853-aa14-be343c592a5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 13:14:13.619603 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:13.619517 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:13.619769 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:13.619517 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:13.619769 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:13.619648 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:13.619769 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:13.619531 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:13.619769 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:13.619736 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:13.619957 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:13.619854 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:15.620269 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:15.619981 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:15.620269 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:15.620055 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:15.620269 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:15.620257 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:15.620814 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:15.620340 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:15.620814 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:15.620076 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:15.620814 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:15.620435 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:17.619962 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:17.619778 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:17.620596 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:17.619786 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:17.620596 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:17.619813 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:17.620596 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:17.620043 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:17.620596 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:17.620163 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:17.620596 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:17.620288 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:17.700553 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:17.700526 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" event={"ID":"4e35f04b-fa9b-4dcf-80fe-d198d261ae2e","Type":"ContainerStarted","Data":"35a65bd1305ba94de9b68c3d4ffbf1561aec45b1bca5e1a6ec52594a8e779e9d"} Jul 02 13:14:17.702244 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:17.702224 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal" event={"ID":"41c092df77332bb9c2b6523d6044f2bf","Type":"ContainerStarted","Data":"22c84d37c4eae0a58457613331540693ecd27ed278efcf87b3bc25990a88ee8a"} Jul 02 13:14:17.705877 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:17.704587 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" event={"ID":"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11","Type":"ContainerStarted","Data":"ee04badca0f1c3b094bc93daa8d400f5bee6a6b71645f9ff97687df3c9966ed8"} Jul 02 13:14:17.705877 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:17.704662 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" event={"ID":"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11","Type":"ContainerStarted","Data":"5309c33d5875d5aaa6bd82271ca0bea3dd3237df9cfe799817f72cfb76aaa063"} Jul 02 13:14:17.705877 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:17.704709 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" event={"ID":"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11","Type":"ContainerStarted","Data":"a08ccc50f83d5c8bc195deae2fc9eda9a8f37adacb403f1992e281eee81c6168"} Jul 02 13:14:17.706857 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:17.706839 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4p9hq" event={"ID":"493bc598-29fb-4757-b9a7-b8bb1b346b77","Type":"ContainerStarted","Data":"4c5df8f07a6e6ce60fa822a309a3e31382b4d60f60065282d9749632108ec393"} Jul 02 13:14:17.724494 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:17.724450 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jv2bn" podStartSLOduration=2.585547672 podStartE2EDuration="21.724437255s" totalImagesPullingTime="19.138889583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:17.72428665 +0000 UTC m=+22.569000329" watchObservedRunningTime="2026-07-02 13:14:17.724437255 +0000 UTC m=+22.569150933" Jul 02 13:14:17.742380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:17.742341 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal" podStartSLOduration=20.742328765 podStartE2EDuration="20.742328765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:17.742112576 +0000 UTC m=+22.586826253" watchObservedRunningTime="2026-07-02 13:14:17.742328765 +0000 UTC m=+22.587042443" Jul 02 13:14:18.710126 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.710047 2578 generic.go:356] "Generic (PLEG): container finished" podID="d935d27e6506ff6c126f79e137292824" containerID="7c331edaef1e972c15905ee47eadd21f85582ea58904ae120475a72ed6184d29" exitCode=0 Jul 02 13:14:18.710761 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.710131 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal" event={"ID":"d935d27e6506ff6c126f79e137292824","Type":"ContainerDied","Data":"7c331edaef1e972c15905ee47eadd21f85582ea58904ae120475a72ed6184d29"} Jul 02 13:14:18.711603 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.711575 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c4k2p" event={"ID":"305c0bc7-b833-4bca-8f4e-6f4a392f09b0","Type":"ContainerStarted","Data":"d4810fcb78f33e4ff85d6fe44e3b6c66277aa232703ca19b1fb2f8f3584d539c"} Jul 02 13:14:18.712963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.712943 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rqfvz" event={"ID":"9a98330d-24e4-490f-810a-bc2488b8e20e","Type":"ContainerStarted","Data":"5bc4752f762bee190c857af905c3dbbd1d2786538663d327cbe89a8cabd76126"} Jul 02 13:14:18.714238 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.714217 2578 generic.go:356] "Generic (PLEG): container finished" podID="d2787138-2777-487c-89da-f039c1f41ad2" containerID="412425ce4e35a60889fcc8adcf35b1afd2649cefbf4bc1247aeea73487b9b732" exitCode=0 Jul 02 13:14:18.714326 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.714289 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkd2" event={"ID":"d2787138-2777-487c-89da-f039c1f41ad2","Type":"ContainerDied","Data":"412425ce4e35a60889fcc8adcf35b1afd2649cefbf4bc1247aeea73487b9b732"} Jul 02 13:14:18.715760 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.715734 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rf9bz" event={"ID":"f9481a84-1265-415b-8a3e-d2c8bfff0ada","Type":"ContainerStarted","Data":"e86b2abc0b21f4627501fa3df50d9ac5c25b468a21fa2ce80379c93c558a5449"} Jul 02 13:14:18.718665 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.718642 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" event={"ID":"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11","Type":"ContainerStarted","Data":"ae1471bc309b3e0d53b6f7d79ddf086e1c524570d6a0aee56d1044306cbb8f81"} Jul 02 13:14:18.718774 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.718701 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" event={"ID":"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11","Type":"ContainerStarted","Data":"1e7346ffa774767e6fd4dcfe857c2702ab671ca73c0967dd154b817cbe98157e"} Jul 02 13:14:18.718834 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.718792 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" event={"ID":"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11","Type":"ContainerStarted","Data":"7c4b46f367bdfb43e0716ee2acdbecea41838d3154974177aca9434764e59aa6"} Jul 02 13:14:18.719959 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.719938 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" event={"ID":"9da8091c-ad4d-4134-9370-11e0e8075f0e","Type":"ContainerStarted","Data":"28621c1bba599b0824754e5903c70d9cb254c1ed4b07de634afb5279e288de06"} Jul 02 13:14:18.721233 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.721215 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qj98s" event={"ID":"bfeff88e-7125-4cf9-a2da-0e30027f22f6","Type":"ContainerStarted","Data":"17ba068441247324e8bbce66ec23abfe7267cfd6c043bd4fd61f6804464c271e"} Jul 02 13:14:18.736263 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.736214 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4p9hq" podStartSLOduration=3.524772574 podStartE2EDuration="22.736195675s" totalImagesPullingTime="19.211423101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:17.763361927 +0000 UTC m=+22.608075616" watchObservedRunningTime="2026-07-02 13:14:18.736195675 +0000 UTC m=+23.580909353" Jul 02 13:14:18.789238 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.789063 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qj98s" podStartSLOduration=3.623122302 podStartE2EDuration="22.789042827s" totalImagesPullingTime="19.165920525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:18.760894799 +0000 UTC m=+23.605608477" watchObservedRunningTime="2026-07-02 13:14:18.789042827 +0000 UTC m=+23.633756505" Jul 02 13:14:18.789740 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.789701 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rf9bz" podStartSLOduration=3.940535696 podStartE2EDuration="22.789691306s" totalImagesPullingTime="18.84915561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:18.789320306 +0000 UTC m=+23.634033965" watchObservedRunningTime="2026-07-02 13:14:18.789691306 +0000 UTC m=+23.634404986" Jul 02 13:14:18.842819 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:18.842776 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rqfvz" podStartSLOduration=4.03339414 podStartE2EDuration="22.842759011s" totalImagesPullingTime="18.809364871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:18.842527552 +0000 UTC m=+23.687241229" watchObservedRunningTime="2026-07-02 13:14:18.842759011 +0000 UTC m=+23.687472690" Jul 02 13:14:19.364301 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:19.364265 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rqfvz" Jul 02 13:14:19.364838 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:19.364824 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rqfvz" Jul 02 13:14:19.392680 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:19.392638 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c4k2p" podStartSLOduration=8.853550745 podStartE2EDuration="23.392621887s" totalImagesPullingTime="14.539071142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:18.874453812 +0000 UTC m=+23.719167489" watchObservedRunningTime="2026-07-02 13:14:19.392621887 +0000 UTC m=+24.237335565" Jul 02 13:14:19.492412 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:19.492385 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 02 13:14:19.619558 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:19.619391 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T13:14:19.492409857Z","UUID":"6e712760-44ad-4b6b-a4cd-2257e427f24a","Handler":null,"Name":"","Endpoint":""} Jul 02 13:14:19.619721 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:19.619641 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:19.619721 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:19.619662 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:19.619804 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:19.619641 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:19.619804 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:19.619771 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:19.619878 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:19.619856 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:19.619956 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:19.619936 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:19.620732 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:19.620710 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 02 13:14:19.620876 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:19.620754 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 02 13:14:19.724822 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:19.724781 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal" event={"ID":"d935d27e6506ff6c126f79e137292824","Type":"ContainerStarted","Data":"f30bdfbcfc8fc59a34f5d1784336ed17d1542807e71e1b927160fa7c0ef88ef5"} Jul 02 13:14:19.726562 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:19.726534 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" event={"ID":"9da8091c-ad4d-4134-9370-11e0e8075f0e","Type":"ContainerStarted","Data":"f7579d9afcee7b629f060c374b8bc4bfe2ea90627dccbc52f91679ba82e8c39e"} Jul 02 13:14:20.731843 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:20.731805 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" event={"ID":"9da8091c-ad4d-4134-9370-11e0e8075f0e","Type":"ContainerStarted","Data":"ea3c11210714bebb947422f8d241fba2b20b481c35bc43fa77ab3a9069b221cf"} Jul 02 13:14:20.735212 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:20.735188 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" event={"ID":"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11","Type":"ContainerStarted","Data":"101e3149a566f49d1d6c92936527f3f0c072ad4d3d75ea348f9bbd40836cb9b6"} Jul 02 13:14:20.735212 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:20.735213 2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 02 13:14:20.767222 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:20.767128 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-243.ec2.internal" podStartSLOduration=23.767104901 podStartE2EDuration="23.767104901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:19.746414281 +0000 UTC m=+24.591127959" watchObservedRunningTime="2026-07-02 13:14:20.767104901 +0000 UTC m=+25.611818577" Jul 02 13:14:20.767425 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:20.767397 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lftgp" podStartSLOduration=7.751430113 podStartE2EDuration="24.767390493s" totalImagesPullingTime="17.01596038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:20.766545704 +0000 UTC m=+25.611259383" watchObservedRunningTime="2026-07-02 13:14:20.767390493 +0000 UTC m=+25.612104172" Jul 02 13:14:21.619170 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:21.619141 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:21.619170 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:21.619158 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:21.619401 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:21.619238 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:21.619401 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:21.619303 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:21.619401 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:21.619324 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:21.619507 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:21.619399 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:22.743863 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:22.743673 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" event={"ID":"3ffbc11c-dcdc-4ff7-b33f-01cd92de5f11","Type":"ContainerStarted","Data":"962893a62df5ca5081c0b5203db12b0dfdf85a25d1686199f91ce59280eb85e6"} Jul 02 13:14:22.744345 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:22.743948 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:14:22.760179 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:22.760155 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:14:22.764318 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:22.764284 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:22.782690 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:22.782624 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" podStartSLOduration=7.307511411 podStartE2EDuration="26.782607511s" totalImagesPullingTime="19.4750961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:22.782570463 +0000 UTC m=+27.627284141" watchObservedRunningTime="2026-07-02 13:14:22.782607511 +0000 UTC m=+27.627321186" Jul 02 13:14:22.884452 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:22.884405 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:22.974222 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:22.974186 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:23.619533 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:23.619493 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:23.620093 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:23.619492 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:23.620093 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:23.619625 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:23.620093 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:23.619626 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:23.620093 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:23.619718 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:23.620093 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:23.619785 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:23.747035 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:23.746954 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:14:23.747035 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:23.747035 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:14:23.764048 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:23.764025 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:14:24.105222 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:24.105182 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:24.550956 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:24.550847 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nmc96"] Jul 02 13:14:24.551126 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:24.551048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:24.551244 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:24.551191 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:24.557370 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:24.557345 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j8m57"] Jul 02 13:14:24.557481 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:24.557442 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g5qbq"] Jul 02 13:14:24.557551 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:24.557490 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:24.557628 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:24.557599 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:24.557752 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:24.557646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:24.557808 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:24.557746 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:24.684818 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:24.684781 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rqfvz" Jul 02 13:14:24.684988 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:24.684954 2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 02 13:14:24.685621 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:24.685596 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rqfvz" Jul 02 13:14:25.620710 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:25.620687 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:25.621225 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:25.620770 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:25.979747 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:25.979713 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:26.093966 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:26.093926 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:26.215411 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:26.215326 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:26.619274 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:26.619246 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:26.619421 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:26.619247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:26.619421 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:26.619331 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:26.619518 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:26.619426 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:26.753449 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:26.753421 2578 generic.go:356] "Generic (PLEG): container finished" podID="d2787138-2777-487c-89da-f039c1f41ad2" containerID="7280b96e44afec33b83569ee01eca762b94f90ab5aac0859bbcb470e752fdf5c" exitCode=0 Jul 02 13:14:26.753781 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:26.753465 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkd2" event={"ID":"d2787138-2777-487c-89da-f039c1f41ad2","Type":"ContainerDied","Data":"7280b96e44afec33b83569ee01eca762b94f90ab5aac0859bbcb470e752fdf5c"} Jul 02 13:14:27.619485 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:27.619456 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:27.619626 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:27.619590 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:28.620014 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:28.619989 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:28.620348 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:28.619989 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:28.620348 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:28.620096 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8m57" podUID="c443c208-241a-4759-a5d9-657a8ff485c2" Jul 02 13:14:28.620348 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:28.620190 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5qbq" podUID="eccf7e61-76e3-4853-aa14-be343c592a5d" Jul 02 13:14:28.760056 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:28.760020 2578 generic.go:356] "Generic (PLEG): container finished" podID="d2787138-2777-487c-89da-f039c1f41ad2" containerID="67d60b30a46d33b30bbf6b5326379b8c0b5d5c59927ed731b54bea3546a00240" exitCode=0 Jul 02 13:14:28.760190 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:28.760087 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkd2" event={"ID":"d2787138-2777-487c-89da-f039c1f41ad2","Type":"ContainerDied","Data":"67d60b30a46d33b30bbf6b5326379b8c0b5d5c59927ed731b54bea3546a00240"} Jul 02 13:14:29.328271 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:29.328231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:29.328409 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:29.328388 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 13:14:29.328459 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:29.328451 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret podName:c443c208-241a-4759-a5d9-657a8ff485c2 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:01.328435236 +0000 UTC m=+66.173148891 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret") pod "global-pull-secret-syncer-j8m57" (UID: "c443c208-241a-4759-a5d9-657a8ff485c2") : object "kube-system"/"original-pull-secret" not registered Jul 02 13:14:29.429099 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:29.429073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs\") pod \"network-metrics-daemon-nmc96\" (UID: \"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4\") " pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:29.429240 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:29.429180 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 13:14:29.429240 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:29.429225 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs podName:ac5d1704-1da6-48e2-b5fa-edecec3a5ee4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:01.429213055 +0000 UTC m=+66.273926711 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs") pod "network-metrics-daemon-nmc96" (UID: "ac5d1704-1da6-48e2-b5fa-edecec3a5ee4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 13:14:29.529756 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:29.529726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frctg\" (UniqueName: \"kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg\") pod \"network-check-target-g5qbq\" (UID: \"eccf7e61-76e3-4853-aa14-be343c592a5d\") " pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:29.529889 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:29.529856 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 13:14:29.529889 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:29.529869 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 13:14:29.529889 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:29.529878 2578 projected.go:196] Error preparing data for projected volume kube-api-access-frctg for pod openshift-network-diagnostics/network-check-target-g5qbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 13:14:29.530014 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:29.529944 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg podName:eccf7e61-76e3-4853-aa14-be343c592a5d nodeName:}" failed. No retries permitted until 2026-07-02 13:15:01.529910801 +0000 UTC m=+66.374624457 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-frctg" (UniqueName: "kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg") pod "network-check-target-g5qbq" (UID: "eccf7e61-76e3-4853-aa14-be343c592a5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 13:14:29.620235 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:29.620171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:29.620631 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:29.620270 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmc96" podUID="ac5d1704-1da6-48e2-b5fa-edecec3a5ee4" Jul 02 13:14:30.044532 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.044368 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-243.ec2.internal" event="NodeReady" Jul 02 13:14:30.044672 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.044630 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 02 13:14:30.180419 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.180362 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8825z"] Jul 02 13:14:30.197128 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.197108 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-87lm7"] Jul 02 13:14:30.197256 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.197235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8825z" Jul 02 13:14:30.205078 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.205062 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 02 13:14:30.205685 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.205672 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 02 13:14:30.206193 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.206179 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5pfr\"" Jul 02 13:14:30.208271 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.208255 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-87lm7" Jul 02 13:14:30.215693 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.215677 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 02 13:14:30.216436 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.216421 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 02 13:14:30.216642 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.216628 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fkpd6\"" Jul 02 13:14:30.217931 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.217898 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 02 13:14:30.232400 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.232379 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8825z"] Jul 02 13:14:30.232976 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.232958 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-87lm7"] Jul 02 13:14:30.334201 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.334176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-tmp-dir\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:30.334316 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.334224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:30.334316 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.334275 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mztx7\" (UniqueName: \"kubernetes.io/projected/c5f79ccc-8051-4f78-b824-b1c8c6652212-kube-api-access-mztx7\") pod \"ingress-canary-87lm7\" (UID: \"c5f79ccc-8051-4f78-b824-b1c8c6652212\") " pod="openshift-ingress-canary/ingress-canary-87lm7" Jul 02 13:14:30.334316 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.334307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-config-volume\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:30.334441 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.334332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5f79ccc-8051-4f78-b824-b1c8c6652212-cert\") pod \"ingress-canary-87lm7\" (UID: \"c5f79ccc-8051-4f78-b824-b1c8c6652212\") " pod="openshift-ingress-canary/ingress-canary-87lm7" Jul 02 13:14:30.334441 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.334353 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5mlv\" (UniqueName: \"kubernetes.io/projected/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-kube-api-access-j5mlv\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:30.435024 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.434971 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-config-volume\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:30.435024 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.435005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5f79ccc-8051-4f78-b824-b1c8c6652212-cert\") pod \"ingress-canary-87lm7\" (UID: \"c5f79ccc-8051-4f78-b824-b1c8c6652212\") " pod="openshift-ingress-canary/ingress-canary-87lm7" Jul 02 13:14:30.435024 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.435022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mlv\" (UniqueName: \"kubernetes.io/projected/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-kube-api-access-j5mlv\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:30.435304 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.435047 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-tmp-dir\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:30.435304 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.435152 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:30.435304 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.435193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mztx7\" (UniqueName: \"kubernetes.io/projected/c5f79ccc-8051-4f78-b824-b1c8c6652212-kube-api-access-mztx7\") pod \"ingress-canary-87lm7\" (UID: \"c5f79ccc-8051-4f78-b824-b1c8c6652212\") " pod="openshift-ingress-canary/ingress-canary-87lm7" Jul 02 13:14:30.435304 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:30.435263 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 13:14:30.435456 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:30.435338 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls podName:fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:30.935308998 +0000 UTC m=+35.780022668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls") pod "dns-default-8825z" (UID: "fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6") : secret "dns-default-metrics-tls" not found Jul 02 13:14:30.435456 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.435376 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-tmp-dir\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:30.435524 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.435506 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-config-volume\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:30.439379 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.439357 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5f79ccc-8051-4f78-b824-b1c8c6652212-cert\") pod \"ingress-canary-87lm7\" (UID: \"c5f79ccc-8051-4f78-b824-b1c8c6652212\") " pod="openshift-ingress-canary/ingress-canary-87lm7" Jul 02 13:14:30.455765 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.455734 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mztx7\" (UniqueName: \"kubernetes.io/projected/c5f79ccc-8051-4f78-b824-b1c8c6652212-kube-api-access-mztx7\") pod \"ingress-canary-87lm7\" (UID: \"c5f79ccc-8051-4f78-b824-b1c8c6652212\") " pod="openshift-ingress-canary/ingress-canary-87lm7" Jul 02 13:14:30.455982 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.455966 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5mlv\" (UniqueName: \"kubernetes.io/projected/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-kube-api-access-j5mlv\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:30.516263 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.516241 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-87lm7" Jul 02 13:14:30.620357 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.620000 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:14:30.620357 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.620144 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:14:30.623828 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.623804 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 13:14:30.623975 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.623964 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zjxr6\"" Jul 02 13:14:30.624462 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.624113 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 13:14:30.624462 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.624296 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 13:14:30.694701 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.694639 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-87lm7"] Jul 02 13:14:30.698510 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:14:30.698487 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f79ccc_8051_4f78_b824_b1c8c6652212.slice/crio-5f602ada71ebb3e650e3c0d1645d0973ca391e829c636eb12aa17424c772cbbb WatchSource:0}: Error finding container 5f602ada71ebb3e650e3c0d1645d0973ca391e829c636eb12aa17424c772cbbb: Status 404 returned error can't find the container with id 5f602ada71ebb3e650e3c0d1645d0973ca391e829c636eb12aa17424c772cbbb Jul 02 13:14:30.771960 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.771935 2578 generic.go:356] "Generic (PLEG): container finished" podID="d2787138-2777-487c-89da-f039c1f41ad2" containerID="5e81be9a14e4d8f611c6f026c9926558a0ad4f4681819595bf8ff7ca6d899f77" exitCode=0 Jul 02 13:14:30.772078 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.772015 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkd2" event={"ID":"d2787138-2777-487c-89da-f039c1f41ad2","Type":"ContainerDied","Data":"5e81be9a14e4d8f611c6f026c9926558a0ad4f4681819595bf8ff7ca6d899f77"} Jul 02 13:14:30.773199 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.773178 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-87lm7" event={"ID":"c5f79ccc-8051-4f78-b824-b1c8c6652212","Type":"ContainerStarted","Data":"5f602ada71ebb3e650e3c0d1645d0973ca391e829c636eb12aa17424c772cbbb"} Jul 02 13:14:30.938067 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:30.938045 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:30.938343 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:30.938316 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 13:14:30.938399 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:30.938389 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls podName:fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:31.938367967 +0000 UTC m=+36.783081638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls") pod "dns-default-8825z" (UID: "fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6") : secret "dns-default-metrics-tls" not found Jul 02 13:14:31.619980 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:31.619944 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:14:31.624106 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:31.624041 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 13:14:31.624557 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:31.624201 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-drw6p\"" Jul 02 13:14:31.946186 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:31.946104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:31.946346 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:31.946272 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 13:14:31.946408 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:31.946350 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls podName:fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:33.946327813 +0000 UTC m=+38.791041471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls") pod "dns-default-8825z" (UID: "fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6") : secret "dns-default-metrics-tls" not found Jul 02 13:14:33.963135 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:33.962885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:33.963487 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:33.963036 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 13:14:33.963487 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:33.963221 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls podName:fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:37.963198778 +0000 UTC m=+42.807912485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls") pod "dns-default-8825z" (UID: "fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6") : secret "dns-default-metrics-tls" not found Jul 02 13:14:35.380327 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:35.380303 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:35.481880 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:35.481813 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:35.784363 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:35.784337 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-87lm7" event={"ID":"c5f79ccc-8051-4f78-b824-b1c8c6652212","Type":"ContainerStarted","Data":"d952eacc044a93cd635d2679cffe88e87d9dc4c188244324d411f746239880d9"} Jul 02 13:14:35.802852 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:35.802808 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-87lm7" podStartSLOduration=1.141163974 podStartE2EDuration="5.802795187s" totalImagesPullingTime="4.661631213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:35.80149797 +0000 UTC m=+40.646211648" watchObservedRunningTime="2026-07-02 13:14:35.802795187 +0000 UTC m=+40.647508859" Jul 02 13:14:36.788975 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:36.788934 2578 generic.go:356] "Generic (PLEG): container finished" podID="d2787138-2777-487c-89da-f039c1f41ad2" containerID="09d439082db82dc3765fdc391d46813269ab11035ea68ec25c39111130168329" exitCode=0 Jul 02 13:14:36.789394 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:36.789009 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkd2" event={"ID":"d2787138-2777-487c-89da-f039c1f41ad2","Type":"ContainerDied","Data":"09d439082db82dc3765fdc391d46813269ab11035ea68ec25c39111130168329"} Jul 02 13:14:37.794235 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:37.794196 2578 generic.go:356] "Generic (PLEG): container finished" podID="d2787138-2777-487c-89da-f039c1f41ad2" containerID="8ebd02cd9d6c16d73825d2140c37c52b40c3ddfc2fcb16cbf1f3462355085f56" exitCode=0 Jul 02 13:14:37.794235 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:37.794238 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkd2" event={"ID":"d2787138-2777-487c-89da-f039c1f41ad2","Type":"ContainerDied","Data":"8ebd02cd9d6c16d73825d2140c37c52b40c3ddfc2fcb16cbf1f3462355085f56"} Jul 02 13:14:37.990091 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:37.990068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:37.990196 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:37.990171 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 13:14:37.990244 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:37.990219 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls podName:fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:45.990202196 +0000 UTC m=+50.834915871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls") pod "dns-default-8825z" (UID: "fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6") : secret "dns-default-metrics-tls" not found Jul 02 13:14:38.799954 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:38.799905 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkd2" event={"ID":"d2787138-2777-487c-89da-f039c1f41ad2","Type":"ContainerStarted","Data":"d0a81ceea20bc7c33af1dd8ddf105351362e005b7ee26e9a4bc41d5499aec2eb"} Jul 02 13:14:38.830438 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:38.830393 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zvkd2" podStartSLOduration=9.603625731 podStartE2EDuration="42.830380875s" totalImagesPullingTime="33.226755144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:38.828465389 +0000 UTC m=+43.673179066" watchObservedRunningTime="2026-07-02 13:14:38.830380875 +0000 UTC m=+43.675094553" Jul 02 13:14:38.969739 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:38.969707 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:39.096255 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:39.096221 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:46.039809 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:46.039776 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:14:46.040191 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:46.039874 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 13:14:46.040191 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:46.039949 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls podName:fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:02.039935098 +0000 UTC m=+66.884648753 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls") pod "dns-default-8825z" (UID: "fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6") : secret "dns-default-metrics-tls" not found Jul 02 13:14:46.207382 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:46.207345 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:46.338707 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:46.338674 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:49.625053 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:49.625017 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:49.812405 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:49.812367 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:50.041568 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:50.041465 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:50.293779 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:50.293700 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:50.409291 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:50.409261 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:50.843290 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:50.843257 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:50.943130 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:50.943097 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:51.496180 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:51.496146 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:51.590629 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:51.590599 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:51.894999 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:51.894961 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:51.966838 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:51.966810 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:55.760359 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:55.760332 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbwhh" Jul 02 13:14:56.095389 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:56.095350 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:56.278012 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:56.277984 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:59.059479 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.059444 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5f67b55555-5qxxr"] Jul 02 13:14:59.062055 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.062039 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.079081 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.079063 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7t44f\"" Jul 02 13:14:59.080182 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.080167 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 02 13:14:59.080182 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.080177 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 02 13:14:59.080260 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.080224 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 02 13:14:59.085306 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.085293 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 02 13:14:59.088264 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.088250 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 02 13:14:59.094724 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.094700 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-5qxxr"] Jul 02 13:14:59.122008 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.121983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e38ece80-57ff-4bbf-a9fe-15721b60478c-serving-cert\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.122112 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.122059 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e38ece80-57ff-4bbf-a9fe-15721b60478c-service-ca-bundle\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.122112 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.122089 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e38ece80-57ff-4bbf-a9fe-15721b60478c-snapshots\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.122224 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.122112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcgts\" (UniqueName: \"kubernetes.io/projected/e38ece80-57ff-4bbf-a9fe-15721b60478c-kube-api-access-kcgts\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.122224 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.122163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e38ece80-57ff-4bbf-a9fe-15721b60478c-tmp\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.122224 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.122191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e38ece80-57ff-4bbf-a9fe-15721b60478c-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.217391 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.217360 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d"] Jul 02 13:14:59.219076 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.219061 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:14:59.223386 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.223360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e38ece80-57ff-4bbf-a9fe-15721b60478c-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.223491 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.223427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e38ece80-57ff-4bbf-a9fe-15721b60478c-serving-cert\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.223840 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.223796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e38ece80-57ff-4bbf-a9fe-15721b60478c-service-ca-bundle\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.223905 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.223878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e38ece80-57ff-4bbf-a9fe-15721b60478c-snapshots\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.224181 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.224152 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgts\" (UniqueName: \"kubernetes.io/projected/e38ece80-57ff-4bbf-a9fe-15721b60478c-kube-api-access-kcgts\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.224330 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.224310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e38ece80-57ff-4bbf-a9fe-15721b60478c-tmp\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.224680 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.224661 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e38ece80-57ff-4bbf-a9fe-15721b60478c-snapshots\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.224869 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.224825 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e38ece80-57ff-4bbf-a9fe-15721b60478c-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.225369 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.225342 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e38ece80-57ff-4bbf-a9fe-15721b60478c-tmp\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.225481 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.225384 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e38ece80-57ff-4bbf-a9fe-15721b60478c-service-ca-bundle\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.227787 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.226443 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 02 13:14:59.227787 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.226669 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 02 13:14:59.227787 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.227266 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 02 13:14:59.227787 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.227665 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6d6tm\"" Jul 02 13:14:59.228137 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.228119 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e38ece80-57ff-4bbf-a9fe-15721b60478c-serving-cert\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.229321 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.229302 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 02 13:14:59.245895 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.245874 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcgts\" (UniqueName: \"kubernetes.io/projected/e38ece80-57ff-4bbf-a9fe-15721b60478c-kube-api-access-kcgts\") pod \"insights-operator-5f67b55555-5qxxr\" (UID: \"e38ece80-57ff-4bbf-a9fe-15721b60478c\") " pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.258531 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.258512 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d"] Jul 02 13:14:59.325950 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.325864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6ec53108-0c5d-4c8b-a563-74ab03c784f5-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:14:59.325950 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.325903 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:14:59.326093 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.325997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrh5\" (UniqueName: \"kubernetes.io/projected/6ec53108-0c5d-4c8b-a563-74ab03c784f5-kube-api-access-fkrh5\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:14:59.370183 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.370162 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-5qxxr" Jul 02 13:14:59.427350 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.427316 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6ec53108-0c5d-4c8b-a563-74ab03c784f5-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:14:59.427504 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.427364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:14:59.427504 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.427426 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrh5\" (UniqueName: \"kubernetes.io/projected/6ec53108-0c5d-4c8b-a563-74ab03c784f5-kube-api-access-fkrh5\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:14:59.427619 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:59.427546 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 13:14:59.427675 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:59.427634 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls podName:6ec53108-0c5d-4c8b-a563-74ab03c784f5 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:59.927610639 +0000 UTC m=+64.772324312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-nd45d" (UID: "6ec53108-0c5d-4c8b-a563-74ab03c784f5") : secret "cluster-monitoring-operator-tls" not found Jul 02 13:14:59.428415 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.428385 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6ec53108-0c5d-4c8b-a563-74ab03c784f5-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:14:59.459196 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.459166 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrh5\" (UniqueName: \"kubernetes.io/projected/6ec53108-0c5d-4c8b-a563-74ab03c784f5-kube-api-access-fkrh5\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:14:59.521719 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.521693 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-5qxxr"] Jul 02 13:14:59.524910 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:14:59.524885 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38ece80_57ff_4bbf_a9fe_15721b60478c.slice/crio-ee362bc3e3063e87238d7fc577979d83206e9f8451a42438251f1c6c1830d44c WatchSource:0}: Error finding container ee362bc3e3063e87238d7fc577979d83206e9f8451a42438251f1c6c1830d44c: Status 404 returned error can't find the container with id ee362bc3e3063e87238d7fc577979d83206e9f8451a42438251f1c6c1830d44c Jul 02 13:14:59.575021 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.574992 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:59.759724 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.759666 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:14:59.848526 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.848504 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-5qxxr" event={"ID":"e38ece80-57ff-4bbf-a9fe-15721b60478c","Type":"ContainerStarted","Data":"ee362bc3e3063e87238d7fc577979d83206e9f8451a42438251f1c6c1830d44c"} Jul 02 13:14:59.930217 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:14:59.930197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:14:59.930309 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:59.930288 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 13:14:59.930347 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:14:59.930333 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls podName:6ec53108-0c5d-4c8b-a563-74ab03c784f5 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:00.930321307 +0000 UTC m=+65.775034962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-nd45d" (UID: "6ec53108-0c5d-4c8b-a563-74ab03c784f5") : secret "cluster-monitoring-operator-tls" not found Jul 02 13:15:00.576300 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:00.576264 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:15:00.696617 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:00.696579 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:15:00.936799 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:00.936716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:15:00.936972 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:00.936872 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 13:15:00.936972 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:00.936960 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls podName:6ec53108-0c5d-4c8b-a563-74ab03c784f5 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:02.936938582 +0000 UTC m=+67.781652244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-nd45d" (UID: "6ec53108-0c5d-4c8b-a563-74ab03c784f5") : secret "cluster-monitoring-operator-tls" not found Jul 02 13:15:01.339798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.339757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:15:01.343022 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.343001 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 13:15:01.352856 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.352833 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c443c208-241a-4759-a5d9-657a8ff485c2-original-pull-secret\") pod \"global-pull-secret-syncer-j8m57\" (UID: \"c443c208-241a-4759-a5d9-657a8ff485c2\") " pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:15:01.440808 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.440767 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs\") pod \"network-metrics-daemon-nmc96\" (UID: \"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4\") " pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:15:01.444420 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.444398 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 13:15:01.451926 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:01.451901 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 02 13:15:01.451998 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:01.451983 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs podName:ac5d1704-1da6-48e2-b5fa-edecec3a5ee4 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:05.451966181 +0000 UTC m=+130.296679837 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs") pod "network-metrics-daemon-nmc96" (UID: "ac5d1704-1da6-48e2-b5fa-edecec3a5ee4") : secret "metrics-daemon-secret" not found Jul 02 13:15:01.532702 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.532615 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8m57" Jul 02 13:15:01.541883 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.541850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frctg\" (UniqueName: \"kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg\") pod \"network-check-target-g5qbq\" (UID: \"eccf7e61-76e3-4853-aa14-be343c592a5d\") " pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:15:01.545169 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.545148 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 13:15:01.555752 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.555730 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 13:15:01.565745 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.565717 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frctg\" (UniqueName: \"kubernetes.io/projected/eccf7e61-76e3-4853-aa14-be343c592a5d-kube-api-access-frctg\") pod \"network-check-target-g5qbq\" (UID: \"eccf7e61-76e3-4853-aa14-be343c592a5d\") " pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:15:01.679903 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.679869 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j8m57"] Jul 02 13:15:01.683125 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:15:01.683082 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc443c208_241a_4759_a5d9_657a8ff485c2.slice/crio-524e3a8f992c735641be9fb1c69befada2c229ee7ac26f470ab7c282a6f6a276 WatchSource:0}: Error finding container 524e3a8f992c735641be9fb1c69befada2c229ee7ac26f470ab7c282a6f6a276: Status 404 returned error can't find the container with id 524e3a8f992c735641be9fb1c69befada2c229ee7ac26f470ab7c282a6f6a276 Jul 02 13:15:01.841890 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.841853 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zjxr6\"" Jul 02 13:15:01.847338 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.847318 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:15:01.854965 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.854938 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-5qxxr" event={"ID":"e38ece80-57ff-4bbf-a9fe-15721b60478c","Type":"ContainerStarted","Data":"dda45e42518447a38d2037bbe095c09c9329823a5fdb01baed99e0f3e9c093c8"} Jul 02 13:15:01.856021 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.855998 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j8m57" event={"ID":"c443c208-241a-4759-a5d9-657a8ff485c2","Type":"ContainerStarted","Data":"524e3a8f992c735641be9fb1c69befada2c229ee7ac26f470ab7c282a6f6a276"} Jul 02 13:15:01.882227 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.882171 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5f67b55555-5qxxr" podStartSLOduration=1.195999571 podStartE2EDuration="2.882150018s" totalImagesPullingTime="1.686150447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:01.881276783 +0000 UTC m=+66.725990485" watchObservedRunningTime="2026-07-02 13:15:01.882150018 +0000 UTC m=+66.726863695" Jul 02 13:15:01.985083 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:01.985055 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g5qbq"] Jul 02 13:15:01.987739 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:15:01.987710 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeccf7e61_76e3_4853_aa14_be343c592a5d.slice/crio-4c851285c33133918e699e55af263df0b369f2f3772fbf201977859eff65b4fa WatchSource:0}: Error finding container 4c851285c33133918e699e55af263df0b369f2f3772fbf201977859eff65b4fa: Status 404 returned error can't find the container with id 4c851285c33133918e699e55af263df0b369f2f3772fbf201977859eff65b4fa Jul 02 13:15:02.032388 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.032354 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw"] Jul 02 13:15:02.035233 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.035212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" Jul 02 13:15:02.038370 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.038355 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 02 13:15:02.038866 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.038849 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 02 13:15:02.039315 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.039298 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 02 13:15:02.039435 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.039419 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nvxwv\"" Jul 02 13:15:02.045838 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.045817 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:15:02.046033 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.045861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn7dh\" (UniqueName: \"kubernetes.io/projected/966752ab-e754-4b20-8e44-d0ef0947debf-kube-api-access-zn7dh\") pod \"cluster-samples-operator-9ddc5ff76-kq6rw\" (UID: \"966752ab-e754-4b20-8e44-d0ef0947debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" Jul 02 13:15:02.046033 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.045894 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-kq6rw\" (UID: \"966752ab-e754-4b20-8e44-d0ef0947debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" Jul 02 13:15:02.046033 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:02.045982 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 13:15:02.046200 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:02.046050 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls podName:fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:34.04603 +0000 UTC m=+98.890743663 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls") pod "dns-default-8825z" (UID: "fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6") : secret "dns-default-metrics-tls" not found Jul 02 13:15:02.049060 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.049038 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw"] Jul 02 13:15:02.147140 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.147038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-kq6rw\" (UID: \"966752ab-e754-4b20-8e44-d0ef0947debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" Jul 02 13:15:02.147140 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.147126 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn7dh\" (UniqueName: \"kubernetes.io/projected/966752ab-e754-4b20-8e44-d0ef0947debf-kube-api-access-zn7dh\") pod \"cluster-samples-operator-9ddc5ff76-kq6rw\" (UID: \"966752ab-e754-4b20-8e44-d0ef0947debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" Jul 02 13:15:02.147367 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:02.147210 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 13:15:02.147367 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:02.147282 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls podName:966752ab-e754-4b20-8e44-d0ef0947debf nodeName:}" failed. No retries permitted until 2026-07-02 13:15:02.647265752 +0000 UTC m=+67.491979407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-kq6rw" (UID: "966752ab-e754-4b20-8e44-d0ef0947debf") : secret "samples-operator-tls" not found Jul 02 13:15:02.157666 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.157641 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn7dh\" (UniqueName: \"kubernetes.io/projected/966752ab-e754-4b20-8e44-d0ef0947debf-kube-api-access-zn7dh\") pod \"cluster-samples-operator-9ddc5ff76-kq6rw\" (UID: \"966752ab-e754-4b20-8e44-d0ef0947debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" Jul 02 13:15:02.650509 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.650460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-kq6rw\" (UID: \"966752ab-e754-4b20-8e44-d0ef0947debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" Jul 02 13:15:02.650697 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:02.650632 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 13:15:02.650785 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:02.650771 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls podName:966752ab-e754-4b20-8e44-d0ef0947debf nodeName:}" failed. No retries permitted until 2026-07-02 13:15:03.650705342 +0000 UTC m=+68.495419000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-kq6rw" (UID: "966752ab-e754-4b20-8e44-d0ef0947debf") : secret "samples-operator-tls" not found Jul 02 13:15:02.859300 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.859268 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g5qbq" event={"ID":"eccf7e61-76e3-4853-aa14-be343c592a5d","Type":"ContainerStarted","Data":"4c851285c33133918e699e55af263df0b369f2f3772fbf201977859eff65b4fa"} Jul 02 13:15:02.947385 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.947300 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l5xd6"] Jul 02 13:15:02.949747 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.949636 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:15:02.951664 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.951643 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:02.953206 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.953179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:15:02.953358 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:02.953338 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 13:15:02.953436 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:02.953423 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls podName:6ec53108-0c5d-4c8b-a563-74ab03c784f5 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:06.953402538 +0000 UTC m=+71.798116194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-nd45d" (UID: "6ec53108-0c5d-4c8b-a563-74ab03c784f5") : secret "cluster-monitoring-operator-tls" not found Jul 02 13:15:02.955391 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.955341 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8kq9n\"" Jul 02 13:15:02.955937 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.955904 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 02 13:15:02.956349 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.956330 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 02 13:15:02.983892 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:02.983861 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l5xd6"] Jul 02 13:15:03.054056 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.054021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07a2217b-3ced-42fd-9849-f3300d67d8b9-crio-socket\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.054250 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.054133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07a2217b-3ced-42fd-9849-f3300d67d8b9-data-volume\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.054250 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.054175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjd5l\" (UniqueName: \"kubernetes.io/projected/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-api-access-fjd5l\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.054349 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.054264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.054349 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.054299 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.061653 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.061628 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-fr8d5"] Jul 02 13:15:03.065225 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.065202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-fr8d5" Jul 02 13:15:03.068830 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.068809 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 02 13:15:03.069020 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.069002 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 02 13:15:03.071430 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.071407 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kgktq\"" Jul 02 13:15:03.087426 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.087382 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-fr8d5"] Jul 02 13:15:03.154941 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.154897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07a2217b-3ced-42fd-9849-f3300d67d8b9-data-volume\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.155128 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.154976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjd5l\" (UniqueName: \"kubernetes.io/projected/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-api-access-fjd5l\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.155128 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.155008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.155128 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.155030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.155128 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.155070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07a2217b-3ced-42fd-9849-f3300d67d8b9-crio-socket\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.155128 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.155098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lkbb\" (UniqueName: \"kubernetes.io/projected/5006a319-17cb-4694-9d11-94f89ad9dfe9-kube-api-access-4lkbb\") pod \"volume-data-source-validator-69d9f679db-fr8d5\" (UID: \"5006a319-17cb-4694-9d11-94f89ad9dfe9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-fr8d5" Jul 02 13:15:03.155394 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:03.155167 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 13:15:03.155394 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:03.155211 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 13:15:03.155394 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:03.155245 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls podName:07a2217b-3ced-42fd-9849-f3300d67d8b9 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:03.655224906 +0000 UTC m=+68.499938565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l5xd6" (UID: "07a2217b-3ced-42fd-9849-f3300d67d8b9") : secret "insights-runtime-extractor-tls" not found Jul 02 13:15:03.155394 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.155248 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07a2217b-3ced-42fd-9849-f3300d67d8b9-data-volume\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.155394 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:03.155266 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm podName:07a2217b-3ced-42fd-9849-f3300d67d8b9 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:03.655255602 +0000 UTC m=+68.499969265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l5xd6" (UID: "07a2217b-3ced-42fd-9849-f3300d67d8b9") : configmap "kube-rbac-proxy" not found Jul 02 13:15:03.155394 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.155294 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07a2217b-3ced-42fd-9849-f3300d67d8b9-crio-socket\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.164923 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.164895 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjd5l\" (UniqueName: \"kubernetes.io/projected/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-api-access-fjd5l\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.256176 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.256087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lkbb\" (UniqueName: \"kubernetes.io/projected/5006a319-17cb-4694-9d11-94f89ad9dfe9-kube-api-access-4lkbb\") pod \"volume-data-source-validator-69d9f679db-fr8d5\" (UID: \"5006a319-17cb-4694-9d11-94f89ad9dfe9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-fr8d5" Jul 02 13:15:03.265890 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.265863 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lkbb\" (UniqueName: \"kubernetes.io/projected/5006a319-17cb-4694-9d11-94f89ad9dfe9-kube-api-access-4lkbb\") pod \"volume-data-source-validator-69d9f679db-fr8d5\" (UID: \"5006a319-17cb-4694-9d11-94f89ad9dfe9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-fr8d5" Jul 02 13:15:03.374014 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.373975 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-fr8d5" Jul 02 13:15:03.532014 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.531931 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-fr8d5"] Jul 02 13:15:03.537109 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:15:03.537080 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5006a319_17cb_4694_9d11_94f89ad9dfe9.slice/crio-30aabc5f755122f1b705cde9f1441a34e15e1336836253fda4285f1b35e5c5ee WatchSource:0}: Error finding container 30aabc5f755122f1b705cde9f1441a34e15e1336836253fda4285f1b35e5c5ee: Status 404 returned error can't find the container with id 30aabc5f755122f1b705cde9f1441a34e15e1336836253fda4285f1b35e5c5ee Jul 02 13:15:03.660094 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.660055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-kq6rw\" (UID: \"966752ab-e754-4b20-8e44-d0ef0947debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" Jul 02 13:15:03.660271 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.660182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.660271 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:03.660211 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 13:15:03.660271 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:03.660262 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 13:15:03.660482 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:03.660278 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls podName:966752ab-e754-4b20-8e44-d0ef0947debf nodeName:}" failed. No retries permitted until 2026-07-02 13:15:05.660256498 +0000 UTC m=+70.504970174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-kq6rw" (UID: "966752ab-e754-4b20-8e44-d0ef0947debf") : secret "samples-operator-tls" not found Jul 02 13:15:03.660482 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:03.660298 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm podName:07a2217b-3ced-42fd-9849-f3300d67d8b9 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:04.66028773 +0000 UTC m=+69.505001385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l5xd6" (UID: "07a2217b-3ced-42fd-9849-f3300d67d8b9") : configmap "kube-rbac-proxy" not found Jul 02 13:15:03.660482 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.660211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:03.660482 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:03.660349 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 13:15:03.660482 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:03.660422 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls podName:07a2217b-3ced-42fd-9849-f3300d67d8b9 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:04.660403104 +0000 UTC m=+69.505116762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l5xd6" (UID: "07a2217b-3ced-42fd-9849-f3300d67d8b9") : secret "insights-runtime-extractor-tls" not found Jul 02 13:15:03.863744 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:03.863705 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-fr8d5" event={"ID":"5006a319-17cb-4694-9d11-94f89ad9dfe9","Type":"ContainerStarted","Data":"30aabc5f755122f1b705cde9f1441a34e15e1336836253fda4285f1b35e5c5ee"} Jul 02 13:15:04.026290 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.026259 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-c7h5s"] Jul 02 13:15:04.029081 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.029048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:04.032101 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.032072 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 02 13:15:04.032228 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.032100 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 02 13:15:04.032277 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.032245 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 02 13:15:04.032327 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.032288 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rxnbt\"" Jul 02 13:15:04.033302 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.033282 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 02 13:15:04.038739 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.037982 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 02 13:15:04.040645 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.040623 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-c7h5s"] Jul 02 13:15:04.064055 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.063846 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c82d88b-a9b9-4af2-b988-c6374ee4a034-trusted-ca\") pod \"console-operator-7fdbdf9c99-c7h5s\" (UID: \"1c82d88b-a9b9-4af2-b988-c6374ee4a034\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:04.064055 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.063896 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c82d88b-a9b9-4af2-b988-c6374ee4a034-serving-cert\") pod \"console-operator-7fdbdf9c99-c7h5s\" (UID: \"1c82d88b-a9b9-4af2-b988-c6374ee4a034\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:04.064055 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.063932 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c82d88b-a9b9-4af2-b988-c6374ee4a034-config\") pod \"console-operator-7fdbdf9c99-c7h5s\" (UID: \"1c82d88b-a9b9-4af2-b988-c6374ee4a034\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:04.064055 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.063966 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5g2k\" (UniqueName: \"kubernetes.io/projected/1c82d88b-a9b9-4af2-b988-c6374ee4a034-kube-api-access-s5g2k\") pod \"console-operator-7fdbdf9c99-c7h5s\" (UID: \"1c82d88b-a9b9-4af2-b988-c6374ee4a034\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:04.164643 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.164548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c82d88b-a9b9-4af2-b988-c6374ee4a034-serving-cert\") pod \"console-operator-7fdbdf9c99-c7h5s\" (UID: \"1c82d88b-a9b9-4af2-b988-c6374ee4a034\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:04.164643 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.164596 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c82d88b-a9b9-4af2-b988-c6374ee4a034-config\") pod \"console-operator-7fdbdf9c99-c7h5s\" (UID: \"1c82d88b-a9b9-4af2-b988-c6374ee4a034\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:04.164643 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.164636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5g2k\" (UniqueName: \"kubernetes.io/projected/1c82d88b-a9b9-4af2-b988-c6374ee4a034-kube-api-access-s5g2k\") pod \"console-operator-7fdbdf9c99-c7h5s\" (UID: \"1c82d88b-a9b9-4af2-b988-c6374ee4a034\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:04.164947 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.164732 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c82d88b-a9b9-4af2-b988-c6374ee4a034-trusted-ca\") pod \"console-operator-7fdbdf9c99-c7h5s\" (UID: \"1c82d88b-a9b9-4af2-b988-c6374ee4a034\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:04.165392 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.165364 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c82d88b-a9b9-4af2-b988-c6374ee4a034-config\") pod \"console-operator-7fdbdf9c99-c7h5s\" (UID: \"1c82d88b-a9b9-4af2-b988-c6374ee4a034\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:04.165691 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.165671 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c82d88b-a9b9-4af2-b988-c6374ee4a034-trusted-ca\") pod \"console-operator-7fdbdf9c99-c7h5s\" (UID: \"1c82d88b-a9b9-4af2-b988-c6374ee4a034\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:04.167334 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.167306 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c82d88b-a9b9-4af2-b988-c6374ee4a034-serving-cert\") pod \"console-operator-7fdbdf9c99-c7h5s\" (UID: \"1c82d88b-a9b9-4af2-b988-c6374ee4a034\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:04.174573 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.174545 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5g2k\" (UniqueName: \"kubernetes.io/projected/1c82d88b-a9b9-4af2-b988-c6374ee4a034-kube-api-access-s5g2k\") pod \"console-operator-7fdbdf9c99-c7h5s\" (UID: \"1c82d88b-a9b9-4af2-b988-c6374ee4a034\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:04.341892 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.341854 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:04.500381 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.500344 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-c7h5s"] Jul 02 13:15:04.505849 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:15:04.505812 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c82d88b_a9b9_4af2_b988_c6374ee4a034.slice/crio-0957341cb018818d03feadb8f24087e805b5260646cd273a97a314a000a8c14e WatchSource:0}: Error finding container 0957341cb018818d03feadb8f24087e805b5260646cd273a97a314a000a8c14e: Status 404 returned error can't find the container with id 0957341cb018818d03feadb8f24087e805b5260646cd273a97a314a000a8c14e Jul 02 13:15:04.610073 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.610046 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rf9bz_f9481a84-1265-415b-8a3e-d2c8bfff0ada/dns-node-resolver/0.log" Jul 02 13:15:04.668745 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.668709 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:04.668950 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.668760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:04.668950 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:04.668880 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 13:15:04.668950 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:04.668939 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 13:15:04.669118 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:04.668964 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm podName:07a2217b-3ced-42fd-9849-f3300d67d8b9 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:06.668943518 +0000 UTC m=+71.513657176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l5xd6" (UID: "07a2217b-3ced-42fd-9849-f3300d67d8b9") : configmap "kube-rbac-proxy" not found Jul 02 13:15:04.669118 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:04.669005 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls podName:07a2217b-3ced-42fd-9849-f3300d67d8b9 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:06.668986397 +0000 UTC m=+71.513700056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l5xd6" (UID: "07a2217b-3ced-42fd-9849-f3300d67d8b9") : secret "insights-runtime-extractor-tls" not found Jul 02 13:15:04.868329 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:04.868288 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" event={"ID":"1c82d88b-a9b9-4af2-b988-c6374ee4a034","Type":"ContainerStarted","Data":"0957341cb018818d03feadb8f24087e805b5260646cd273a97a314a000a8c14e"} Jul 02 13:15:05.212327 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.212253 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c4k2p_305c0bc7-b833-4bca-8f4e-6f4a392f09b0/node-ca/0.log" Jul 02 13:15:05.495496 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.495389 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d8596d4c8-scv9b"] Jul 02 13:15:05.498730 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.498709 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.503059 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.503038 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 02 13:15:05.503059 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.503045 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 02 13:15:05.503427 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.503408 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 02 13:15:05.503427 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.503419 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d7kgk\"" Jul 02 13:15:05.514791 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.514771 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 02 13:15:05.519224 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.519203 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8596d4c8-scv9b"] Jul 02 13:15:05.578726 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.578692 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cde15a2-72c5-4dfb-9522-131e148f2a90-ca-trust-extracted\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.578877 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.578746 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cde15a2-72c5-4dfb-9522-131e148f2a90-trusted-ca\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.578877 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.578856 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmdqs\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-kube-api-access-tmdqs\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.579002 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.578932 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cde15a2-72c5-4dfb-9522-131e148f2a90-image-registry-private-configuration\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.579002 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.578960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-certificates\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.579002 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.578992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.579143 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.579097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-bound-sa-token\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.579227 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.579197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cde15a2-72c5-4dfb-9522-131e148f2a90-installation-pull-secrets\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.679980 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.679942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.680135 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.680012 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-bound-sa-token\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.680135 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:05.680114 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 13:15:05.680135 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:05.680132 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8596d4c8-scv9b: secret "image-registry-tls" not found Jul 02 13:15:05.680263 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.680187 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-kq6rw\" (UID: \"966752ab-e754-4b20-8e44-d0ef0947debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" Jul 02 13:15:05.680263 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:05.680194 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls podName:2cde15a2-72c5-4dfb-9522-131e148f2a90 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:06.180177237 +0000 UTC m=+71.024890892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls") pod "image-registry-5d8596d4c8-scv9b" (UID: "2cde15a2-72c5-4dfb-9522-131e148f2a90") : secret "image-registry-tls" not found Jul 02 13:15:05.680351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.680283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cde15a2-72c5-4dfb-9522-131e148f2a90-installation-pull-secrets\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.680351 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:05.680308 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 13:15:05.680424 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:05.680355 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls podName:966752ab-e754-4b20-8e44-d0ef0947debf nodeName:}" failed. No retries permitted until 2026-07-02 13:15:09.680342402 +0000 UTC m=+74.525056058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-kq6rw" (UID: "966752ab-e754-4b20-8e44-d0ef0947debf") : secret "samples-operator-tls" not found Jul 02 13:15:05.680424 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.680308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cde15a2-72c5-4dfb-9522-131e148f2a90-ca-trust-extracted\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.680424 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.680401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cde15a2-72c5-4dfb-9522-131e148f2a90-trusted-ca\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.680529 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.680428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmdqs\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-kube-api-access-tmdqs\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.680529 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.680456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cde15a2-72c5-4dfb-9522-131e148f2a90-image-registry-private-configuration\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.680661 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.680641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-certificates\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.680746 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.680725 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cde15a2-72c5-4dfb-9522-131e148f2a90-ca-trust-extracted\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.681232 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.681209 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-certificates\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.681445 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.681423 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cde15a2-72c5-4dfb-9522-131e148f2a90-trusted-ca\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.683241 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.683210 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cde15a2-72c5-4dfb-9522-131e148f2a90-image-registry-private-configuration\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.683342 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.683210 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cde15a2-72c5-4dfb-9522-131e148f2a90-installation-pull-secrets\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.696991 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.696961 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-bound-sa-token\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.697190 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.697172 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmdqs\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-kube-api-access-tmdqs\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:05.810628 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.810597 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-87lm7_c5f79ccc-8051-4f78-b824-b1c8c6652212/serve-healthcheck-canary/0.log" Jul 02 13:15:05.872811 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.872770 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-fr8d5" event={"ID":"5006a319-17cb-4694-9d11-94f89ad9dfe9","Type":"ContainerStarted","Data":"dd6a75ef8d404375e0931c081b05a3af27fd0ccc845f1cbd9909f4e0dad721b7"} Jul 02 13:15:05.993189 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:05.993151 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:15:06.083136 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:06.083041 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" limit=1024 Jul 02 13:15:06.183762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:06.183719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:06.183974 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:06.183888 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 13:15:06.183974 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:06.183910 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8596d4c8-scv9b: secret "image-registry-tls" not found Jul 02 13:15:06.184086 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:06.183999 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls podName:2cde15a2-72c5-4dfb-9522-131e148f2a90 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:07.18397641 +0000 UTC m=+72.028690099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls") pod "image-registry-5d8596d4c8-scv9b" (UID: "2cde15a2-72c5-4dfb-9522-131e148f2a90") : secret "image-registry-tls" not found Jul 02 13:15:06.687899 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:06.687856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:06.688113 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:06.687906 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:06.688113 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:06.688032 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 13:15:06.688113 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:06.688044 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 13:15:06.688113 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:06.688098 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm podName:07a2217b-3ced-42fd-9849-f3300d67d8b9 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:10.688079661 +0000 UTC m=+75.532793336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l5xd6" (UID: "07a2217b-3ced-42fd-9849-f3300d67d8b9") : configmap "kube-rbac-proxy" not found Jul 02 13:15:06.688319 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:06.688119 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls podName:07a2217b-3ced-42fd-9849-f3300d67d8b9 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:10.688110328 +0000 UTC m=+75.532823983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l5xd6" (UID: "07a2217b-3ced-42fd-9849-f3300d67d8b9") : secret "insights-runtime-extractor-tls" not found Jul 02 13:15:06.990649 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:06.990554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:15:06.991113 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:06.990698 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 13:15:06.991113 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:06.990791 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls podName:6ec53108-0c5d-4c8b-a563-74ab03c784f5 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:14.990764315 +0000 UTC m=+79.835477988 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-nd45d" (UID: "6ec53108-0c5d-4c8b-a563-74ab03c784f5") : secret "cluster-monitoring-operator-tls" not found Jul 02 13:15:07.192595 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:07.192552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:07.192780 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:07.192710 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 13:15:07.192780 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:07.192733 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8596d4c8-scv9b: secret "image-registry-tls" not found Jul 02 13:15:07.192897 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:07.192806 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls podName:2cde15a2-72c5-4dfb-9522-131e148f2a90 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:09.192782635 +0000 UTC m=+74.037496295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls") pod "image-registry-5d8596d4c8-scv9b" (UID: "2cde15a2-72c5-4dfb-9522-131e148f2a90") : secret "image-registry-tls" not found Jul 02 13:15:08.881965 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:08.881912 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j8m57" event={"ID":"c443c208-241a-4759-a5d9-657a8ff485c2","Type":"ContainerStarted","Data":"bb984ad6e001e5f5f6df186bb51f53e6aa4c644907ceac14ab68c027c87d43d3"} Jul 02 13:15:08.883295 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:08.883275 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g5qbq" event={"ID":"eccf7e61-76e3-4853-aa14-be343c592a5d","Type":"ContainerStarted","Data":"07e56e95303751b258b556af59125edfba4699bcdfbd6538aac8f550f4f19c0c"} Jul 02 13:15:08.883390 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:08.883368 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:15:08.884687 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:08.884669 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/0.log" Jul 02 13:15:08.884788 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:08.884703 2578 generic.go:356] "Generic (PLEG): container finished" podID="1c82d88b-a9b9-4af2-b988-c6374ee4a034" containerID="a6e81c3e6fd63227ad05d3b05df75ad5ef150a66557d7196aeebd6b8416fdd18" exitCode=255 Jul 02 13:15:08.884788 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:08.884726 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" event={"ID":"1c82d88b-a9b9-4af2-b988-c6374ee4a034","Type":"ContainerDied","Data":"a6e81c3e6fd63227ad05d3b05df75ad5ef150a66557d7196aeebd6b8416fdd18"} Jul 02 13:15:08.884939 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:08.884910 2578 scope.go:117] "RemoveContainer" containerID="a6e81c3e6fd63227ad05d3b05df75ad5ef150a66557d7196aeebd6b8416fdd18" Jul 02 13:15:08.899780 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:08.899735 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j8m57" podStartSLOduration=66.314306972 podStartE2EDuration="1m12.899719977s" totalImagesPullingTime="6.585413005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:08.899509824 +0000 UTC m=+73.744223499" watchObservedRunningTime="2026-07-02 13:15:08.899719977 +0000 UTC m=+73.744433656" Jul 02 13:15:08.900471 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:08.900440 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-fr8d5" podStartSLOduration=4.5150884730000005 podStartE2EDuration="5.900426239s" totalImagesPullingTime="1.385337766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:05.889530052 +0000 UTC m=+70.734243756" watchObservedRunningTime="2026-07-02 13:15:08.900426239 +0000 UTC m=+73.745139918" Jul 02 13:15:08.919995 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:08.919953 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g5qbq" podStartSLOduration=66.650500431 podStartE2EDuration="1m12.919940385s" totalImagesPullingTime="6.269439954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:08.919541699 +0000 UTC m=+73.764255406" watchObservedRunningTime="2026-07-02 13:15:08.919940385 +0000 UTC m=+73.764654062" Jul 02 13:15:09.208655 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:09.208575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:09.208800 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:09.208719 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 13:15:09.208800 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:09.208733 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8596d4c8-scv9b: secret "image-registry-tls" not found Jul 02 13:15:09.208800 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:09.208790 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls podName:2cde15a2-72c5-4dfb-9522-131e148f2a90 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:13.208771992 +0000 UTC m=+78.053485671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls") pod "image-registry-5d8596d4c8-scv9b" (UID: "2cde15a2-72c5-4dfb-9522-131e148f2a90") : secret "image-registry-tls" not found Jul 02 13:15:09.711189 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:09.711147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-kq6rw\" (UID: \"966752ab-e754-4b20-8e44-d0ef0947debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" Jul 02 13:15:09.711360 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:09.711296 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 13:15:09.711401 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:09.711363 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls podName:966752ab-e754-4b20-8e44-d0ef0947debf nodeName:}" failed. No retries permitted until 2026-07-02 13:15:17.711343954 +0000 UTC m=+82.556057618 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-kq6rw" (UID: "966752ab-e754-4b20-8e44-d0ef0947debf") : secret "samples-operator-tls" not found Jul 02 13:15:09.889510 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:09.889481 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/1.log" Jul 02 13:15:09.889884 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:09.889853 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/0.log" Jul 02 13:15:09.889964 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:09.889885 2578 generic.go:356] "Generic (PLEG): container finished" podID="1c82d88b-a9b9-4af2-b988-c6374ee4a034" containerID="297c4684bcd8c60e27b7bbccccee9d1639eea63f08e14227941b4bed0001fb89" exitCode=255 Jul 02 13:15:09.889964 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:09.889928 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" event={"ID":"1c82d88b-a9b9-4af2-b988-c6374ee4a034","Type":"ContainerDied","Data":"297c4684bcd8c60e27b7bbccccee9d1639eea63f08e14227941b4bed0001fb89"} Jul 02 13:15:09.890036 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:09.890011 2578 scope.go:117] "RemoveContainer" containerID="a6e81c3e6fd63227ad05d3b05df75ad5ef150a66557d7196aeebd6b8416fdd18" Jul 02 13:15:09.890284 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:09.890266 2578 scope.go:117] "RemoveContainer" containerID="297c4684bcd8c60e27b7bbccccee9d1639eea63f08e14227941b4bed0001fb89" Jul 02 13:15:09.890482 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:09.890460 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-c7h5s_openshift-console-operator(1c82d88b-a9b9-4af2-b988-c6374ee4a034)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" podUID="1c82d88b-a9b9-4af2-b988-c6374ee4a034" Jul 02 13:15:10.719174 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:10.719138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:10.719174 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:10.719177 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:10.719386 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:10.719255 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 13:15:10.719386 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:10.719302 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm podName:07a2217b-3ced-42fd-9849-f3300d67d8b9 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:18.71928867 +0000 UTC m=+83.564002326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l5xd6" (UID: "07a2217b-3ced-42fd-9849-f3300d67d8b9") : configmap "kube-rbac-proxy" not found Jul 02 13:15:10.719386 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:10.719339 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 13:15:10.719482 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:10.719399 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls podName:07a2217b-3ced-42fd-9849-f3300d67d8b9 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:18.719385048 +0000 UTC m=+83.564098704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l5xd6" (UID: "07a2217b-3ced-42fd-9849-f3300d67d8b9") : secret "insights-runtime-extractor-tls" not found Jul 02 13:15:10.757815 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:10.757788 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-s59jf"] Jul 02 13:15:10.760320 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:10.760305 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-s59jf" Jul 02 13:15:10.763303 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:10.763284 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rxkvc\"" Jul 02 13:15:10.769652 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:10.769632 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-s59jf"] Jul 02 13:15:10.819573 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:10.819545 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvtjs\" (UniqueName: \"kubernetes.io/projected/085f7da4-425c-4c12-a45b-96b460711663-kube-api-access-cvtjs\") pod \"network-check-source-56499946bc-s59jf\" (UID: \"085f7da4-425c-4c12-a45b-96b460711663\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-s59jf" Jul 02 13:15:10.894213 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:10.894192 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/1.log" Jul 02 13:15:10.894554 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:10.894480 2578 scope.go:117] "RemoveContainer" containerID="297c4684bcd8c60e27b7bbccccee9d1639eea63f08e14227941b4bed0001fb89" Jul 02 13:15:10.894643 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:10.894623 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-c7h5s_openshift-console-operator(1c82d88b-a9b9-4af2-b988-c6374ee4a034)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" podUID="1c82d88b-a9b9-4af2-b988-c6374ee4a034" Jul 02 13:15:10.920778 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:10.920751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtjs\" (UniqueName: \"kubernetes.io/projected/085f7da4-425c-4c12-a45b-96b460711663-kube-api-access-cvtjs\") pod \"network-check-source-56499946bc-s59jf\" (UID: \"085f7da4-425c-4c12-a45b-96b460711663\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-s59jf" Jul 02 13:15:10.930098 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:10.930079 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtjs\" (UniqueName: \"kubernetes.io/projected/085f7da4-425c-4c12-a45b-96b460711663-kube-api-access-cvtjs\") pod \"network-check-source-56499946bc-s59jf\" (UID: \"085f7da4-425c-4c12-a45b-96b460711663\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-s59jf" Jul 02 13:15:11.068415 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:11.068386 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-s59jf" Jul 02 13:15:11.194114 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:11.194081 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-s59jf"] Jul 02 13:15:11.197342 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:15:11.197320 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085f7da4_425c_4c12_a45b_96b460711663.slice/crio-242b3162157017c1307593650db4d37d7878c44d34ac78e260c62ed8b0ad7aeb WatchSource:0}: Error finding container 242b3162157017c1307593650db4d37d7878c44d34ac78e260c62ed8b0ad7aeb: Status 404 returned error can't find the container with id 242b3162157017c1307593650db4d37d7878c44d34ac78e260c62ed8b0ad7aeb Jul 02 13:15:11.900362 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:11.900329 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-s59jf" event={"ID":"085f7da4-425c-4c12-a45b-96b460711663","Type":"ContainerStarted","Data":"2621e2439ff55dd4a94adfff022d2da51bd1ced9093684843d5207f857c67e88"} Jul 02 13:15:11.900821 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:11.900375 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-s59jf" event={"ID":"085f7da4-425c-4c12-a45b-96b460711663","Type":"ContainerStarted","Data":"242b3162157017c1307593650db4d37d7878c44d34ac78e260c62ed8b0ad7aeb"} Jul 02 13:15:11.922962 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:11.922891 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-56499946bc-s59jf" podStartSLOduration=1.92287641 podStartE2EDuration="1.92287641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:11.922700572 +0000 UTC m=+76.767414263" watchObservedRunningTime="2026-07-02 13:15:11.92287641 +0000 UTC m=+76.767590107" Jul 02 13:15:13.239613 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:13.239568 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:13.240009 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:13.239681 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 13:15:13.240009 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:13.239692 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8596d4c8-scv9b: secret "image-registry-tls" not found Jul 02 13:15:13.240009 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:13.239741 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls podName:2cde15a2-72c5-4dfb-9522-131e148f2a90 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:21.239726612 +0000 UTC m=+86.084440268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls") pod "image-registry-5d8596d4c8-scv9b" (UID: "2cde15a2-72c5-4dfb-9522-131e148f2a90") : secret "image-registry-tls" not found Jul 02 13:15:14.343159 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:14.343125 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:14.343540 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:14.343182 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:14.343540 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:14.343483 2578 scope.go:117] "RemoveContainer" containerID="297c4684bcd8c60e27b7bbccccee9d1639eea63f08e14227941b4bed0001fb89" Jul 02 13:15:14.343630 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:14.343610 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-c7h5s_openshift-console-operator(1c82d88b-a9b9-4af2-b988-c6374ee4a034)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" podUID="1c82d88b-a9b9-4af2-b988-c6374ee4a034" Jul 02 13:15:15.056179 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:15.056134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:15:15.056366 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:15.056282 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 13:15:15.056366 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:15.056354 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls podName:6ec53108-0c5d-4c8b-a563-74ab03c784f5 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:31.056337628 +0000 UTC m=+95.901051291 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-nd45d" (UID: "6ec53108-0c5d-4c8b-a563-74ab03c784f5") : secret "cluster-monitoring-operator-tls" not found Jul 02 13:15:16.133956 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.133903 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-znjvk"] Jul 02 13:15:16.137253 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.137237 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-znjvk" Jul 02 13:15:16.140753 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.140729 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 02 13:15:16.141790 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.141774 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 02 13:15:16.141881 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.141777 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 02 13:15:16.141881 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.141777 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6zntl\"" Jul 02 13:15:16.141881 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.141827 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 02 13:15:16.160772 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.160750 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-znjvk"] Jul 02 13:15:16.265753 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.265723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6ad7dff8-ca07-4f5c-b921-b7d4e00689c9-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-znjvk\" (UID: \"6ad7dff8-ca07-4f5c-b921-b7d4e00689c9\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-znjvk" Jul 02 13:15:16.265912 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.265762 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtq62\" (UniqueName: \"kubernetes.io/projected/6ad7dff8-ca07-4f5c-b921-b7d4e00689c9-kube-api-access-gtq62\") pod \"service-ca-7bb4f4bcc7-znjvk\" (UID: \"6ad7dff8-ca07-4f5c-b921-b7d4e00689c9\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-znjvk" Jul 02 13:15:16.265912 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.265792 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6ad7dff8-ca07-4f5c-b921-b7d4e00689c9-signing-key\") pod \"service-ca-7bb4f4bcc7-znjvk\" (UID: \"6ad7dff8-ca07-4f5c-b921-b7d4e00689c9\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-znjvk" Jul 02 13:15:16.366431 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.366383 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6ad7dff8-ca07-4f5c-b921-b7d4e00689c9-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-znjvk\" (UID: \"6ad7dff8-ca07-4f5c-b921-b7d4e00689c9\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-znjvk" Jul 02 13:15:16.366431 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.366441 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtq62\" (UniqueName: \"kubernetes.io/projected/6ad7dff8-ca07-4f5c-b921-b7d4e00689c9-kube-api-access-gtq62\") pod \"service-ca-7bb4f4bcc7-znjvk\" (UID: \"6ad7dff8-ca07-4f5c-b921-b7d4e00689c9\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-znjvk" Jul 02 13:15:16.366574 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.366544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6ad7dff8-ca07-4f5c-b921-b7d4e00689c9-signing-key\") pod \"service-ca-7bb4f4bcc7-znjvk\" (UID: \"6ad7dff8-ca07-4f5c-b921-b7d4e00689c9\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-znjvk" Jul 02 13:15:16.367247 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.367224 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6ad7dff8-ca07-4f5c-b921-b7d4e00689c9-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-znjvk\" (UID: \"6ad7dff8-ca07-4f5c-b921-b7d4e00689c9\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-znjvk" Jul 02 13:15:16.368901 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.368879 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6ad7dff8-ca07-4f5c-b921-b7d4e00689c9-signing-key\") pod \"service-ca-7bb4f4bcc7-znjvk\" (UID: \"6ad7dff8-ca07-4f5c-b921-b7d4e00689c9\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-znjvk" Jul 02 13:15:16.376905 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.376881 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtq62\" (UniqueName: \"kubernetes.io/projected/6ad7dff8-ca07-4f5c-b921-b7d4e00689c9-kube-api-access-gtq62\") pod \"service-ca-7bb4f4bcc7-znjvk\" (UID: \"6ad7dff8-ca07-4f5c-b921-b7d4e00689c9\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-znjvk" Jul 02 13:15:16.445686 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.445622 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-znjvk" Jul 02 13:15:16.571537 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.571507 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-znjvk"] Jul 02 13:15:16.574675 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:15:16.574648 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad7dff8_ca07_4f5c_b921_b7d4e00689c9.slice/crio-7a110581bbe2b003114e7212da5ad4b96de21df9b09665e54198e5f16e0de4e5 WatchSource:0}: Error finding container 7a110581bbe2b003114e7212da5ad4b96de21df9b09665e54198e5f16e0de4e5: Status 404 returned error can't find the container with id 7a110581bbe2b003114e7212da5ad4b96de21df9b09665e54198e5f16e0de4e5 Jul 02 13:15:16.915386 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:16.915352 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-znjvk" event={"ID":"6ad7dff8-ca07-4f5c-b921-b7d4e00689c9","Type":"ContainerStarted","Data":"7a110581bbe2b003114e7212da5ad4b96de21df9b09665e54198e5f16e0de4e5"} Jul 02 13:15:17.778120 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:17.778089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-kq6rw\" (UID: \"966752ab-e754-4b20-8e44-d0ef0947debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" Jul 02 13:15:17.778559 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:17.778216 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 13:15:17.778559 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:17.778293 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls podName:966752ab-e754-4b20-8e44-d0ef0947debf nodeName:}" failed. No retries permitted until 2026-07-02 13:15:33.778272451 +0000 UTC m=+98.622986123 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-kq6rw" (UID: "966752ab-e754-4b20-8e44-d0ef0947debf") : secret "samples-operator-tls" not found Jul 02 13:15:18.786804 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:18.786758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:18.786804 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:18.786809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:18.787281 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:18.786904 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 13:15:18.787281 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:18.786986 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls podName:07a2217b-3ced-42fd-9849-f3300d67d8b9 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:34.786970446 +0000 UTC m=+99.631684101 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l5xd6" (UID: "07a2217b-3ced-42fd-9849-f3300d67d8b9") : secret "insights-runtime-extractor-tls" not found Jul 02 13:15:18.787368 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:18.787349 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07a2217b-3ced-42fd-9849-f3300d67d8b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:19.925779 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:19.925685 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-znjvk" event={"ID":"6ad7dff8-ca07-4f5c-b921-b7d4e00689c9","Type":"ContainerStarted","Data":"bc74e3a876e3f45468c3f9cc0944898145f763ce8eb29993f663443422594128"} Jul 02 13:15:19.948367 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:19.948315 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7bb4f4bcc7-znjvk" podStartSLOduration=0.877057434 podStartE2EDuration="3.948299024s" totalImagesPullingTime="3.07124159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:19.946527234 +0000 UTC m=+84.791240912" watchObservedRunningTime="2026-07-02 13:15:19.948299024 +0000 UTC m=+84.793012731" Jul 02 13:15:21.309043 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:21.309000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls\") pod \"image-registry-5d8596d4c8-scv9b\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:21.309514 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:21.309124 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 13:15:21.309514 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:21.309138 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8596d4c8-scv9b: secret "image-registry-tls" not found Jul 02 13:15:21.309514 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:21.309184 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls podName:2cde15a2-72c5-4dfb-9522-131e148f2a90 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:37.30917151 +0000 UTC m=+102.153885166 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls") pod "image-registry-5d8596d4c8-scv9b" (UID: "2cde15a2-72c5-4dfb-9522-131e148f2a90") : secret "image-registry-tls" not found Jul 02 13:15:27.620001 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:27.619971 2578 scope.go:117] "RemoveContainer" containerID="297c4684bcd8c60e27b7bbccccee9d1639eea63f08e14227941b4bed0001fb89" Jul 02 13:15:27.951817 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:27.951764 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:15:27.952185 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:27.952158 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/1.log" Jul 02 13:15:27.952297 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:27.952193 2578 generic.go:356] "Generic (PLEG): container finished" podID="1c82d88b-a9b9-4af2-b988-c6374ee4a034" containerID="2c672ee62a134f5bd94c9425143eb6c892f354e88d2350d4ad594db97b97a5a1" exitCode=255 Jul 02 13:15:27.952297 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:27.952231 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" event={"ID":"1c82d88b-a9b9-4af2-b988-c6374ee4a034","Type":"ContainerDied","Data":"2c672ee62a134f5bd94c9425143eb6c892f354e88d2350d4ad594db97b97a5a1"} Jul 02 13:15:27.952297 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:27.952287 2578 scope.go:117] "RemoveContainer" containerID="297c4684bcd8c60e27b7bbccccee9d1639eea63f08e14227941b4bed0001fb89" Jul 02 13:15:27.952615 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:27.952586 2578 scope.go:117] "RemoveContainer" containerID="2c672ee62a134f5bd94c9425143eb6c892f354e88d2350d4ad594db97b97a5a1" Jul 02 13:15:27.952795 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:27.952775 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-c7h5s_openshift-console-operator(1c82d88b-a9b9-4af2-b988-c6374ee4a034)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" podUID="1c82d88b-a9b9-4af2-b988-c6374ee4a034" Jul 02 13:15:28.955953 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:28.955911 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:15:31.084753 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:31.084717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:15:31.087066 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:31.087040 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ec53108-0c5d-4c8b-a563-74ab03c784f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-nd45d\" (UID: \"6ec53108-0c5d-4c8b-a563-74ab03c784f5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:15:31.328848 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:31.328818 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" Jul 02 13:15:31.461210 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:31.461182 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d"] Jul 02 13:15:31.464727 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:15:31.464701 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec53108_0c5d_4c8b_a563_74ab03c784f5.slice/crio-9b1abed4b3b077599e84a5fbed252e1a00e731214774252b578bfb6b4d55b922 WatchSource:0}: Error finding container 9b1abed4b3b077599e84a5fbed252e1a00e731214774252b578bfb6b4d55b922: Status 404 returned error can't find the container with id 9b1abed4b3b077599e84a5fbed252e1a00e731214774252b578bfb6b4d55b922 Jul 02 13:15:31.965512 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:31.965479 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" event={"ID":"6ec53108-0c5d-4c8b-a563-74ab03c784f5","Type":"ContainerStarted","Data":"9b1abed4b3b077599e84a5fbed252e1a00e731214774252b578bfb6b4d55b922"} Jul 02 13:15:33.805140 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:33.805104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-kq6rw\" (UID: \"966752ab-e754-4b20-8e44-d0ef0947debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" Jul 02 13:15:33.807423 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:33.807395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/966752ab-e754-4b20-8e44-d0ef0947debf-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-kq6rw\" (UID: \"966752ab-e754-4b20-8e44-d0ef0947debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" Jul 02 13:15:33.844409 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:33.844383 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" Jul 02 13:15:34.001064 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.001034 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw"] Jul 02 13:15:34.107628 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.107599 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:15:34.109717 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.109691 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6-metrics-tls\") pod \"dns-default-8825z\" (UID: \"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6\") " pod="openshift-dns/dns-default-8825z" Jul 02 13:15:34.342485 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.342454 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:34.342608 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.342521 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:34.342798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.342774 2578 scope.go:117] "RemoveContainer" containerID="2c672ee62a134f5bd94c9425143eb6c892f354e88d2350d4ad594db97b97a5a1" Jul 02 13:15:34.342992 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:34.342971 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-c7h5s_openshift-console-operator(1c82d88b-a9b9-4af2-b988-c6374ee4a034)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" podUID="1c82d88b-a9b9-4af2-b988-c6374ee4a034" Jul 02 13:15:34.411504 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.411475 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5pfr\"" Jul 02 13:15:34.419141 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.419118 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8825z" Jul 02 13:15:34.545986 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.545965 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8825z"] Jul 02 13:15:34.547827 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:15:34.547798 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa757129_f6f3_4ce7_8bb1_79dfe3cc47a6.slice/crio-2134cd794aeeedc0bf6421c0dd4dad1274fcea2e3c1beb50d4661f1ca500d037 WatchSource:0}: Error finding container 2134cd794aeeedc0bf6421c0dd4dad1274fcea2e3c1beb50d4661f1ca500d037: Status 404 returned error can't find the container with id 2134cd794aeeedc0bf6421c0dd4dad1274fcea2e3c1beb50d4661f1ca500d037 Jul 02 13:15:34.812399 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.812364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:34.814988 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.814964 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07a2217b-3ced-42fd-9849-f3300d67d8b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5xd6\" (UID: \"07a2217b-3ced-42fd-9849-f3300d67d8b9\") " pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:34.976558 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.976506 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8825z" event={"ID":"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6","Type":"ContainerStarted","Data":"2134cd794aeeedc0bf6421c0dd4dad1274fcea2e3c1beb50d4661f1ca500d037"} Jul 02 13:15:34.978227 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.978192 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" event={"ID":"6ec53108-0c5d-4c8b-a563-74ab03c784f5","Type":"ContainerStarted","Data":"5ed5220eed457e8cc4424958c0241335aa629e3da9ceb417127789f884a531c1"} Jul 02 13:15:34.979527 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.979505 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" event={"ID":"966752ab-e754-4b20-8e44-d0ef0947debf","Type":"ContainerStarted","Data":"460af89dc1b5cc03a6e5b29e3777f9103dfef6070621b65957bb0c7abbf9ebe4"} Jul 02 13:15:34.979812 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:34.979795 2578 scope.go:117] "RemoveContainer" containerID="2c672ee62a134f5bd94c9425143eb6c892f354e88d2350d4ad594db97b97a5a1" Jul 02 13:15:34.979979 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:34.979960 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-c7h5s_openshift-console-operator(1c82d88b-a9b9-4af2-b988-c6374ee4a034)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" podUID="1c82d88b-a9b9-4af2-b988-c6374ee4a034" Jul 02 13:15:35.004655 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.004613 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-nd45d" podStartSLOduration=33.53977047 podStartE2EDuration="36.004599922s" totalImagesPullingTime="2.464829452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:35.003958623 +0000 UTC m=+99.848672303" watchObservedRunningTime="2026-07-02 13:15:35.004599922 +0000 UTC m=+99.849313597" Jul 02 13:15:35.063683 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.063606 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l5xd6" Jul 02 13:15:35.235670 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:15:35.235626 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a2217b_3ced_42fd_9849_f3300d67d8b9.slice/crio-cc6b51444a908acad256b18311c7fc2e0ec2c25036be66918422fbe44663b19a WatchSource:0}: Error finding container cc6b51444a908acad256b18311c7fc2e0ec2c25036be66918422fbe44663b19a: Status 404 returned error can't find the container with id cc6b51444a908acad256b18311c7fc2e0ec2c25036be66918422fbe44663b19a Jul 02 13:15:35.243079 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.243044 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l5xd6"] Jul 02 13:15:35.729421 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.729391 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2"] Jul 02 13:15:35.735486 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.735466 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:35.747037 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:35.746975 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"klusterlet-addon-workmgr-5575c65698-6l6d2\" is forbidden: User \"system:node:ip-10-0-131-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-243.ec2.internal' and this object" podUID="452570e9-d618-4898-8d0a-85b68043ce2c" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:35.747594 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:35.747570 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"work-manager-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-131-243.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-243.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" type="*v1.Secret" Jul 02 13:15:35.747722 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:35.747601 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-131-243.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-243.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 02 13:15:35.748695 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:35.748659 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"open-cluster-management-image-pull-credentials\" is forbidden: User \"system:node:ip-10-0-131-243.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-243.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" Jul 02 13:15:35.748973 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:35.748950 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-131-243.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-243.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 02 13:15:35.775813 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.775781 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb"] Jul 02 13:15:35.783181 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.783156 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2"] Jul 02 13:15:35.783337 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.783302 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:35.792802 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.791953 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 02 13:15:35.792802 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.792175 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 02 13:15:35.794561 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.794540 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 02 13:15:35.794935 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.794904 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 02 13:15:35.808409 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.808386 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb"] Jul 02 13:15:35.809556 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.809537 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d8596d4c8-scv9b"] Jul 02 13:15:35.809784 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:35.809761 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" podUID="2cde15a2-72c5-4dfb-9522-131e148f2a90" Jul 02 13:15:35.821061 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.821034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/452570e9-d618-4898-8d0a-85b68043ce2c-klusterlet-config\") pod \"klusterlet-addon-workmgr-5575c65698-6l6d2\" (UID: \"452570e9-d618-4898-8d0a-85b68043ce2c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:35.821457 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.821098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c5p4\" (UniqueName: \"kubernetes.io/projected/452570e9-d618-4898-8d0a-85b68043ce2c-kube-api-access-5c5p4\") pod \"klusterlet-addon-workmgr-5575c65698-6l6d2\" (UID: \"452570e9-d618-4898-8d0a-85b68043ce2c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:35.821457 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.821178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/452570e9-d618-4898-8d0a-85b68043ce2c-tmp\") pod \"klusterlet-addon-workmgr-5575c65698-6l6d2\" (UID: \"452570e9-d618-4898-8d0a-85b68043ce2c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:35.922525 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.922491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:35.922525 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.922540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/452570e9-d618-4898-8d0a-85b68043ce2c-tmp\") pod \"klusterlet-addon-workmgr-5575c65698-6l6d2\" (UID: \"452570e9-d618-4898-8d0a-85b68043ce2c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:35.922736 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.922567 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-ca\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:35.922736 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.922615 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-hub\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:35.922736 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.922673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:35.922736 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.922701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zbkm\" (UniqueName: \"kubernetes.io/projected/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-kube-api-access-4zbkm\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:35.922736 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.922727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/452570e9-d618-4898-8d0a-85b68043ce2c-klusterlet-config\") pod \"klusterlet-addon-workmgr-5575c65698-6l6d2\" (UID: \"452570e9-d618-4898-8d0a-85b68043ce2c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:35.922948 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.922771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:35.922948 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.922805 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c5p4\" (UniqueName: \"kubernetes.io/projected/452570e9-d618-4898-8d0a-85b68043ce2c-kube-api-access-5c5p4\") pod \"klusterlet-addon-workmgr-5575c65698-6l6d2\" (UID: \"452570e9-d618-4898-8d0a-85b68043ce2c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:35.922948 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.922837 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/452570e9-d618-4898-8d0a-85b68043ce2c-tmp\") pod \"klusterlet-addon-workmgr-5575c65698-6l6d2\" (UID: \"452570e9-d618-4898-8d0a-85b68043ce2c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:35.949783 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.949755 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8f89cb6f4-mhk48"] Jul 02 13:15:35.953056 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.953033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:35.977503 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.977481 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8f89cb6f4-mhk48"] Jul 02 13:15:35.987878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.987794 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l5xd6" event={"ID":"07a2217b-3ced-42fd-9849-f3300d67d8b9","Type":"ContainerStarted","Data":"cc6b51444a908acad256b18311c7fc2e0ec2c25036be66918422fbe44663b19a"} Jul 02 13:15:35.987878 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.987823 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:35.994141 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:35.994121 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:36.023764 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.023734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:36.023894 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.023780 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b1379fe-4206-4604-a25a-b036a50f866f-registry-certificates\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.023894 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.023802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-ca\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:36.023894 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.023864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-hub\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:36.024076 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.023900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b1379fe-4206-4604-a25a-b036a50f866f-installation-pull-secrets\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.024076 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.023941 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9ms5\" (UniqueName: \"kubernetes.io/projected/4b1379fe-4206-4604-a25a-b036a50f866f-kube-api-access-d9ms5\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.024076 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.023975 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:36.024076 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.023997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zbkm\" (UniqueName: \"kubernetes.io/projected/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-kube-api-access-4zbkm\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:36.024076 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.024019 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b1379fe-4206-4604-a25a-b036a50f866f-bound-sa-token\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.024076 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.024044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b1379fe-4206-4604-a25a-b036a50f866f-image-registry-private-configuration\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.024384 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.024111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:36.024384 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.024143 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b1379fe-4206-4604-a25a-b036a50f866f-trusted-ca\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.024384 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.024175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b1379fe-4206-4604-a25a-b036a50f866f-ca-trust-extracted\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.024384 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.024209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b1379fe-4206-4604-a25a-b036a50f866f-registry-tls\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.026601 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.026579 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:36.026707 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.026581 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-ca\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:36.026707 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.026666 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:36.026935 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.026896 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-hub\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:36.125441 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.125405 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-certificates\") pod \"2cde15a2-72c5-4dfb-9522-131e148f2a90\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " Jul 02 13:15:36.125627 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.125488 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cde15a2-72c5-4dfb-9522-131e148f2a90-image-registry-private-configuration\") pod \"2cde15a2-72c5-4dfb-9522-131e148f2a90\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " Jul 02 13:15:36.125627 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.125515 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cde15a2-72c5-4dfb-9522-131e148f2a90-trusted-ca\") pod \"2cde15a2-72c5-4dfb-9522-131e148f2a90\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " Jul 02 13:15:36.125627 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.125550 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmdqs\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-kube-api-access-tmdqs\") pod \"2cde15a2-72c5-4dfb-9522-131e148f2a90\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " Jul 02 13:15:36.125627 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.125586 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cde15a2-72c5-4dfb-9522-131e148f2a90-ca-trust-extracted\") pod \"2cde15a2-72c5-4dfb-9522-131e148f2a90\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " Jul 02 13:15:36.125627 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.125614 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cde15a2-72c5-4dfb-9522-131e148f2a90-installation-pull-secrets\") pod \"2cde15a2-72c5-4dfb-9522-131e148f2a90\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " Jul 02 13:15:36.125871 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.125644 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-bound-sa-token\") pod \"2cde15a2-72c5-4dfb-9522-131e148f2a90\" (UID: \"2cde15a2-72c5-4dfb-9522-131e148f2a90\") " Jul 02 13:15:36.125871 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.125755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b1379fe-4206-4604-a25a-b036a50f866f-installation-pull-secrets\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.125871 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.125783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9ms5\" (UniqueName: \"kubernetes.io/projected/4b1379fe-4206-4604-a25a-b036a50f866f-kube-api-access-d9ms5\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.125871 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.125821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b1379fe-4206-4604-a25a-b036a50f866f-bound-sa-token\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.125871 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.125820 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2cde15a2-72c5-4dfb-9522-131e148f2a90" (UID: "2cde15a2-72c5-4dfb-9522-131e148f2a90"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 13:15:36.126147 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.126101 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cde15a2-72c5-4dfb-9522-131e148f2a90-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2cde15a2-72c5-4dfb-9522-131e148f2a90" (UID: "2cde15a2-72c5-4dfb-9522-131e148f2a90"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:15:36.126361 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.126323 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cde15a2-72c5-4dfb-9522-131e148f2a90-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2cde15a2-72c5-4dfb-9522-131e148f2a90" (UID: "2cde15a2-72c5-4dfb-9522-131e148f2a90"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 13:15:36.126451 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.126370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b1379fe-4206-4604-a25a-b036a50f866f-image-registry-private-configuration\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.126503 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.126458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b1379fe-4206-4604-a25a-b036a50f866f-trusted-ca\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.126559 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.126501 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b1379fe-4206-4604-a25a-b036a50f866f-ca-trust-extracted\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.126559 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.126536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b1379fe-4206-4604-a25a-b036a50f866f-registry-tls\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.127857 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.126657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b1379fe-4206-4604-a25a-b036a50f866f-registry-certificates\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.127857 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.126752 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cde15a2-72c5-4dfb-9522-131e148f2a90-ca-trust-extracted\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:15:36.127857 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.126767 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-certificates\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:15:36.127857 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.126783 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cde15a2-72c5-4dfb-9522-131e148f2a90-trusted-ca\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:15:36.127857 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.127018 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b1379fe-4206-4604-a25a-b036a50f866f-ca-trust-extracted\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.127857 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.127535 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b1379fe-4206-4604-a25a-b036a50f866f-registry-certificates\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.128623 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.128598 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2cde15a2-72c5-4dfb-9522-131e148f2a90" (UID: "2cde15a2-72c5-4dfb-9522-131e148f2a90"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 13:15:36.128744 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.128627 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cde15a2-72c5-4dfb-9522-131e148f2a90-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2cde15a2-72c5-4dfb-9522-131e148f2a90" (UID: "2cde15a2-72c5-4dfb-9522-131e148f2a90"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 13:15:36.128744 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.128723 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cde15a2-72c5-4dfb-9522-131e148f2a90-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2cde15a2-72c5-4dfb-9522-131e148f2a90" (UID: "2cde15a2-72c5-4dfb-9522-131e148f2a90"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 13:15:36.129088 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.129062 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b1379fe-4206-4604-a25a-b036a50f866f-trusted-ca\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.129300 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.129264 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b1379fe-4206-4604-a25a-b036a50f866f-image-registry-private-configuration\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.129385 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.129366 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b1379fe-4206-4604-a25a-b036a50f866f-installation-pull-secrets\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.129492 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.129476 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-kube-api-access-tmdqs" (OuterVolumeSpecName: "kube-api-access-tmdqs") pod "2cde15a2-72c5-4dfb-9522-131e148f2a90" (UID: "2cde15a2-72c5-4dfb-9522-131e148f2a90"). InnerVolumeSpecName "kube-api-access-tmdqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 13:15:36.130353 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.130336 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b1379fe-4206-4604-a25a-b036a50f866f-registry-tls\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.135109 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.135088 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9ms5\" (UniqueName: \"kubernetes.io/projected/4b1379fe-4206-4604-a25a-b036a50f866f-kube-api-access-d9ms5\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.136633 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.136613 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b1379fe-4206-4604-a25a-b036a50f866f-bound-sa-token\") pod \"image-registry-8f89cb6f4-mhk48\" (UID: \"4b1379fe-4206-4604-a25a-b036a50f866f\") " pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.227744 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.227714 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmdqs\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-kube-api-access-tmdqs\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:15:36.227744 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.227739 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cde15a2-72c5-4dfb-9522-131e148f2a90-installation-pull-secrets\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:15:36.227744 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.227749 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-bound-sa-token\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:15:36.227980 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.227758 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cde15a2-72c5-4dfb-9522-131e148f2a90-image-registry-private-configuration\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:15:36.262797 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.262712 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.524157 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.524107 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8f89cb6f4-mhk48"] Jul 02 13:15:36.527866 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:15:36.527840 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1379fe_4206_4604_a25a_b036a50f866f.slice/crio-2ed0c2c7eb1297b50f1e73b1f2413a38631dea20f87a5545bc897301024ab300 WatchSource:0}: Error finding container 2ed0c2c7eb1297b50f1e73b1f2413a38631dea20f87a5545bc897301024ab300: Status 404 returned error can't find the container with id 2ed0c2c7eb1297b50f1e73b1f2413a38631dea20f87a5545bc897301024ab300 Jul 02 13:15:36.667590 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.667362 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 02 13:15:36.782767 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.782688 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 02 13:15:36.923506 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:36.923465 2578 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/work-manager-hub-kubeconfig: failed to sync secret cache: timed out waiting for the condition Jul 02 13:15:36.923888 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:36.923549 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/452570e9-d618-4898-8d0a-85b68043ce2c-klusterlet-config podName:452570e9-d618-4898-8d0a-85b68043ce2c nodeName:}" failed. No retries permitted until 2026-07-02 13:15:37.423532662 +0000 UTC m=+102.268246318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "klusterlet-config" (UniqueName: "kubernetes.io/secret/452570e9-d618-4898-8d0a-85b68043ce2c-klusterlet-config") pod "klusterlet-addon-workmgr-5575c65698-6l6d2" (UID: "452570e9-d618-4898-8d0a-85b68043ce2c") : failed to sync secret cache: timed out waiting for the condition Jul 02 13:15:36.941389 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.941364 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 02 13:15:36.945421 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.945399 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:36.949577 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.949554 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zbkm\" (UniqueName: \"kubernetes.io/projected/485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9-kube-api-access-4zbkm\") pod \"cluster-proxy-proxy-agent-9c96f895b-q22sb\" (UID: \"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:36.953230 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.953210 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c5p4\" (UniqueName: \"kubernetes.io/projected/452570e9-d618-4898-8d0a-85b68043ce2c-kube-api-access-5c5p4\") pod \"klusterlet-addon-workmgr-5575c65698-6l6d2\" (UID: \"452570e9-d618-4898-8d0a-85b68043ce2c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:36.993233 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.993204 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" event={"ID":"966752ab-e754-4b20-8e44-d0ef0947debf","Type":"ContainerStarted","Data":"04e0174c3c37b1c4c0ed470ee233b086bebfc5728f4197c8e8a322322bc41b58"} Jul 02 13:15:36.993356 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.993256 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" event={"ID":"966752ab-e754-4b20-8e44-d0ef0947debf","Type":"ContainerStarted","Data":"b278933195661ac8c8d30dc1eb0b0d1b651cd5e91295f8d278f848f17059786c"} Jul 02 13:15:36.994662 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.994636 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" event={"ID":"4b1379fe-4206-4604-a25a-b036a50f866f","Type":"ContainerStarted","Data":"4b855ce3a828267e29ea39643bd91d18a82fce2c67ef29c5d999352964c9600b"} Jul 02 13:15:36.994762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.994679 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" event={"ID":"4b1379fe-4206-4604-a25a-b036a50f866f","Type":"ContainerStarted","Data":"2ed0c2c7eb1297b50f1e73b1f2413a38631dea20f87a5545bc897301024ab300"} Jul 02 13:15:36.994762 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.994706 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:15:36.996264 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.996248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8596d4c8-scv9b" Jul 02 13:15:36.996264 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.996248 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8825z" event={"ID":"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6","Type":"ContainerStarted","Data":"ef2d7d1bbb0efcbac8f71a2fcbc685fa57db8e806f93395a0ae3ba9d3ebc4d6a"} Jul 02 13:15:36.996437 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:36.996291 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8825z" event={"ID":"fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6","Type":"ContainerStarted","Data":"fba197f4b93bb4ffd79f1c48fadee89a8f7b39d70b7324e54afada94d8f6c2e7"} Jul 02 13:15:37.005366 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:37.005344 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" Jul 02 13:15:37.015570 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:37.015531 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-kq6rw" podStartSLOduration=32.708702618 podStartE2EDuration="35.015516911s" totalImagesPullingTime="2.306814293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:37.01448409 +0000 UTC m=+101.859197766" watchObservedRunningTime="2026-07-02 13:15:37.015516911 +0000 UTC m=+101.860230589" Jul 02 13:15:37.040235 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:37.040117 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8825z" podStartSLOduration=65.17575755 podStartE2EDuration="1m7.040096089s" totalImagesPullingTime="1.864338539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:37.038745618 +0000 UTC m=+101.883459288" watchObservedRunningTime="2026-07-02 13:15:37.040096089 +0000 UTC m=+101.884809746" Jul 02 13:15:37.066834 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:37.066788 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" podStartSLOduration=2.066767431 podStartE2EDuration="2.066767431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:37.066401111 +0000 UTC m=+101.911114789" watchObservedRunningTime="2026-07-02 13:15:37.066767431 +0000 UTC m=+101.911481109" Jul 02 13:15:37.106443 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:37.106394 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d8596d4c8-scv9b"] Jul 02 13:15:37.119280 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:37.119208 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d8596d4c8-scv9b"] Jul 02 13:15:37.152461 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:37.152437 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb"] Jul 02 13:15:37.156050 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:15:37.156023 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod485ebe24_0930_4ffe_bdc9_6fa3c13fb9c9.slice/crio-944aab75e3ac689d98a7e0b2d84b610e892c817faa1cb2b3de335e93b38be9cc WatchSource:0}: Error finding container 944aab75e3ac689d98a7e0b2d84b610e892c817faa1cb2b3de335e93b38be9cc: Status 404 returned error can't find the container with id 944aab75e3ac689d98a7e0b2d84b610e892c817faa1cb2b3de335e93b38be9cc Jul 02 13:15:37.235213 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:37.235181 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 02 13:15:37.237593 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:37.237572 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cde15a2-72c5-4dfb-9522-131e148f2a90-registry-tls\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:15:37.439154 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:37.439072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/452570e9-d618-4898-8d0a-85b68043ce2c-klusterlet-config\") pod \"klusterlet-addon-workmgr-5575c65698-6l6d2\" (UID: \"452570e9-d618-4898-8d0a-85b68043ce2c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:37.443350 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:37.443323 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/452570e9-d618-4898-8d0a-85b68043ce2c-klusterlet-config\") pod \"klusterlet-addon-workmgr-5575c65698-6l6d2\" (UID: \"452570e9-d618-4898-8d0a-85b68043ce2c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:37.547154 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:37.546884 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:37.637977 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:37.637200 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cde15a2-72c5-4dfb-9522-131e148f2a90" path="/var/lib/kubelet/pods/2cde15a2-72c5-4dfb-9522-131e148f2a90/volumes" Jul 02 13:15:37.727621 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:37.727462 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2"] Jul 02 13:15:37.731400 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:15:37.731366 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452570e9_d618_4898_8d0a_85b68043ce2c.slice/crio-1045bc35a7a47c88f0432297a60395f494274cc1886577bb3e244bb45cbf1be7 WatchSource:0}: Error finding container 1045bc35a7a47c88f0432297a60395f494274cc1886577bb3e244bb45cbf1be7: Status 404 returned error can't find the container with id 1045bc35a7a47c88f0432297a60395f494274cc1886577bb3e244bb45cbf1be7 Jul 02 13:15:38.001910 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:38.001834 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" event={"ID":"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9","Type":"ContainerStarted","Data":"944aab75e3ac689d98a7e0b2d84b610e892c817faa1cb2b3de335e93b38be9cc"} Jul 02 13:15:38.003102 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:38.003067 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" event={"ID":"452570e9-d618-4898-8d0a-85b68043ce2c","Type":"ContainerStarted","Data":"1045bc35a7a47c88f0432297a60395f494274cc1886577bb3e244bb45cbf1be7"} Jul 02 13:15:38.003777 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:38.003746 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8825z" Jul 02 13:15:39.893025 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:39.893000 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g5qbq" Jul 02 13:15:45.029269 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:45.029234 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l5xd6" event={"ID":"07a2217b-3ced-42fd-9849-f3300d67d8b9","Type":"ContainerStarted","Data":"0b34375ceac0e5463be6833a1369fd7da7dd1d224766faf8e8157313a4256892"} Jul 02 13:15:45.031024 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:45.031001 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" event={"ID":"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9","Type":"ContainerStarted","Data":"22b29dad68b2693715745bfd1cc1575aef6ffb0e22a660bddb5d4ae8e3722ce4"} Jul 02 13:15:45.031159 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:45.031034 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" event={"ID":"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9","Type":"ContainerStarted","Data":"f626e56eb262a09e83d0eff6146e60b363cee8c52c74a3e4891d21833b7ea0ec"} Jul 02 13:15:45.031159 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:45.031046 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" event={"ID":"485ebe24-0930-4ffe-bdc9-6fa3c13fb9c9","Type":"ContainerStarted","Data":"95ac0f314b0d299c8921f845aee88a389bf1247cb8ef610944eb50efd6d6ecc6"} Jul 02 13:15:45.032459 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:45.032434 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" event={"ID":"452570e9-d618-4898-8d0a-85b68043ce2c","Type":"ContainerStarted","Data":"1cbb1ba1771193f53546fc79e407deb837253cbc30b9a61f3e7e489bb6957cca"} Jul 02 13:15:45.032681 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:45.032658 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:45.034178 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:45.034162 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" Jul 02 13:15:45.064167 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:45.064129 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c96f895b-q22sb" podStartSLOduration=2.921136009 podStartE2EDuration="10.064115911s" totalImagesPullingTime="7.142979902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:45.063773841 +0000 UTC m=+109.908487552" watchObservedRunningTime="2026-07-02 13:15:45.064115911 +0000 UTC m=+109.908829589" Jul 02 13:15:46.037127 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:46.037085 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l5xd6" event={"ID":"07a2217b-3ced-42fd-9849-f3300d67d8b9","Type":"ContainerStarted","Data":"76fc84014da06b47207f42ca2c1930ee08dbbc136feb73ac756b6a6db99fe210"} Jul 02 13:15:48.009381 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:48.009354 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8825z" Jul 02 13:15:48.030529 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:48.030480 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5575c65698-6l6d2" podStartSLOduration=6.449327011 podStartE2EDuration="13.030464978s" totalImagesPullingTime="6.581137967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:45.087473031 +0000 UTC m=+109.932186705" watchObservedRunningTime="2026-07-02 13:15:48.030464978 +0000 UTC m=+112.875178651" Jul 02 13:15:49.620658 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:49.620629 2578 scope.go:117] "RemoveContainer" containerID="2c672ee62a134f5bd94c9425143eb6c892f354e88d2350d4ad594db97b97a5a1" Jul 02 13:15:50.054572 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.054549 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:15:50.054684 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.054635 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" event={"ID":"1c82d88b-a9b9-4af2-b988-c6374ee4a034","Type":"ContainerStarted","Data":"3f526a58708deb5dd50b52bad97cd09e3dd5f33a668b24dde70bf6fd8273f28f"} Jul 02 13:15:50.054994 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.054945 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:50.056048 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.056023 2578 patch_prober.go:28] interesting pod/console-operator-7fdbdf9c99-c7h5s container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.13:8443/readyz\": dial tcp 10.133.0.13:8443: connect: connection refused" start-of-body= Jul 02 13:15:50.056179 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.056147 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" podUID="1c82d88b-a9b9-4af2-b988-c6374ee4a034" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.13:8443/readyz\": dial tcp 10.133.0.13:8443: connect: connection refused" Jul 02 13:15:50.057042 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.057019 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l5xd6" event={"ID":"07a2217b-3ced-42fd-9849-f3300d67d8b9","Type":"ContainerStarted","Data":"79c048aa37eb59125629d1a3b19e6853920713da17fe5db8313f0af812944de4"} Jul 02 13:15:50.122963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.122856 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" podStartSLOduration=42.325637991 podStartE2EDuration="46.122841809s" totalImagesPullingTime="3.797203818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:50.08532341 +0000 UTC m=+114.930037086" watchObservedRunningTime="2026-07-02 13:15:50.122841809 +0000 UTC m=+114.967555485" Jul 02 13:15:50.123637 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.123601 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l5xd6" podStartSLOduration=33.511692657 podStartE2EDuration="48.123588965s" totalImagesPullingTime="14.611896308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:15:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:50.116087318 +0000 UTC m=+114.960801024" watchObservedRunningTime="2026-07-02 13:15:50.123588965 +0000 UTC m=+114.968302643" Jul 02 13:15:50.308686 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.308604 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v2jxh"] Jul 02 13:15:50.310843 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.310827 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.316102 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.316082 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 02 13:15:50.316544 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.316518 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 02 13:15:50.316795 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.316773 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 02 13:15:50.317798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.317544 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f4hzn\"" Jul 02 13:15:50.322037 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.322021 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 02 13:15:50.349405 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.349385 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41a0736c-8423-45ec-a26b-e8ba78af9526-sys\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.349500 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.349417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-wtmp\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.349500 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.349445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.349619 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.349596 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-textfile\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.349664 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.349635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkpz6\" (UniqueName: \"kubernetes.io/projected/41a0736c-8423-45ec-a26b-e8ba78af9526-kube-api-access-dkpz6\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.349706 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.349679 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.349742 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.349707 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-tls\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.349742 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.349730 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41a0736c-8423-45ec-a26b-e8ba78af9526-metrics-client-ca\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.349839 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.349783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41a0736c-8423-45ec-a26b-e8ba78af9526-root\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.450639 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.450608 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41a0736c-8423-45ec-a26b-e8ba78af9526-sys\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.450825 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.450801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-wtmp\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.450932 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.450835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.451000 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.450932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-textfile\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.451000 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.450957 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkpz6\" (UniqueName: \"kubernetes.io/projected/41a0736c-8423-45ec-a26b-e8ba78af9526-kube-api-access-dkpz6\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.451098 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.451000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.451098 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.451029 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-tls\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.451098 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.451052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41a0736c-8423-45ec-a26b-e8ba78af9526-metrics-client-ca\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.451098 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.451088 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41a0736c-8423-45ec-a26b-e8ba78af9526-root\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.451305 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.451183 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41a0736c-8423-45ec-a26b-e8ba78af9526-root\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.451305 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.450720 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41a0736c-8423-45ec-a26b-e8ba78af9526-sys\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.451398 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.451351 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-wtmp\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.451932 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:50.451578 2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 02 13:15:50.451932 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:15:50.451659 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-tls podName:41a0736c-8423-45ec-a26b-e8ba78af9526 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:50.951638153 +0000 UTC m=+115.796351810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-tls") pod "node-exporter-v2jxh" (UID: "41a0736c-8423-45ec-a26b-e8ba78af9526") : secret "node-exporter-tls" not found Jul 02 13:15:50.452144 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.451962 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-textfile\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.452288 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.452257 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.452803 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.452788 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41a0736c-8423-45ec-a26b-e8ba78af9526-metrics-client-ca\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.453753 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.453708 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.467289 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.467244 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkpz6\" (UniqueName: \"kubernetes.io/projected/41a0736c-8423-45ec-a26b-e8ba78af9526-kube-api-access-dkpz6\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.955629 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.955594 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-tls\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:50.957981 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:50.957950 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41a0736c-8423-45ec-a26b-e8ba78af9526-node-exporter-tls\") pod \"node-exporter-v2jxh\" (UID: \"41a0736c-8423-45ec-a26b-e8ba78af9526\") " pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:51.063893 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:51.063863 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-c7h5s" Jul 02 13:15:51.220990 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:51.220883 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v2jxh" Jul 02 13:15:51.231468 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:15:51.231441 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a0736c_8423_45ec_a26b_e8ba78af9526.slice/crio-496e366d478ae88060c1ae32cd187440198d48bd0e74cba3b4aeb321d0e8dfe5 WatchSource:0}: Error finding container 496e366d478ae88060c1ae32cd187440198d48bd0e74cba3b4aeb321d0e8dfe5: Status 404 returned error can't find the container with id 496e366d478ae88060c1ae32cd187440198d48bd0e74cba3b4aeb321d0e8dfe5 Jul 02 13:15:52.065334 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:52.065305 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2jxh" event={"ID":"41a0736c-8423-45ec-a26b-e8ba78af9526","Type":"ContainerStarted","Data":"496e366d478ae88060c1ae32cd187440198d48bd0e74cba3b4aeb321d0e8dfe5"} Jul 02 13:15:53.071191 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:53.071154 2578 generic.go:356] "Generic (PLEG): container finished" podID="41a0736c-8423-45ec-a26b-e8ba78af9526" containerID="db59af7ba66c1010cc2ba7fb72fd3e0d14e53698c106b022575a7f9d2aa6e369" exitCode=0 Jul 02 13:15:53.071665 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:53.071199 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2jxh" event={"ID":"41a0736c-8423-45ec-a26b-e8ba78af9526","Type":"ContainerDied","Data":"db59af7ba66c1010cc2ba7fb72fd3e0d14e53698c106b022575a7f9d2aa6e369"} Jul 02 13:15:54.076279 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:54.076243 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2jxh" event={"ID":"41a0736c-8423-45ec-a26b-e8ba78af9526","Type":"ContainerStarted","Data":"4a2e048794e95dd089c8c45f8dcb4b4a0ea22954452b2631b4c92e068666c717"} Jul 02 13:15:54.076745 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:54.076295 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2jxh" event={"ID":"41a0736c-8423-45ec-a26b-e8ba78af9526","Type":"ContainerStarted","Data":"fd0687b1f6164f0c534f20a5d1a782d2b27b5e76f476eeb58aa49a97fd175df5"} Jul 02 13:15:54.101109 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:54.101063 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v2jxh" podStartSLOduration=2.917217849 podStartE2EDuration="4.101049232s" totalImagesPullingTime="1.119898045s" totalInitContainerRuntime="63.933338ms" isStatefulPod=true podCreationTimestamp="2026-07-02 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:54.099349174 +0000 UTC m=+118.944062872" watchObservedRunningTime="2026-07-02 13:15:54.101049232 +0000 UTC m=+118.945762924" Jul 02 13:15:56.266806 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:56.266777 2578 patch_prober.go:28] interesting pod/image-registry-8f89cb6f4-mhk48 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 02 13:15:56.267179 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:56.266824 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" podUID="4b1379fe-4206-4604-a25a-b036a50f866f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 13:15:58.007772 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:15:58.007741 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8f89cb6f4-mhk48" Jul 02 13:16:05.468716 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:05.468670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs\") pod \"network-metrics-daemon-nmc96\" (UID: \"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4\") " pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:16:05.471084 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:05.471057 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5d1704-1da6-48e2-b5fa-edecec3a5ee4-metrics-certs\") pod \"network-metrics-daemon-nmc96\" (UID: \"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4\") " pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:16:05.533892 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:05.533867 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-drw6p\"" Jul 02 13:16:05.541479 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:05.541458 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmc96" Jul 02 13:16:05.694359 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:05.694326 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nmc96"] Jul 02 13:16:05.697628 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:16:05.697597 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5d1704_1da6_48e2_b5fa_edecec3a5ee4.slice/crio-90220f5c1bc7cc35c16e4aa983e7319a47716747cc490c785491ad53154286d2 WatchSource:0}: Error finding container 90220f5c1bc7cc35c16e4aa983e7319a47716747cc490c785491ad53154286d2: Status 404 returned error can't find the container with id 90220f5c1bc7cc35c16e4aa983e7319a47716747cc490c785491ad53154286d2 Jul 02 13:16:06.116755 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:06.116722 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmc96" event={"ID":"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4","Type":"ContainerStarted","Data":"90220f5c1bc7cc35c16e4aa983e7319a47716747cc490c785491ad53154286d2"} Jul 02 13:16:07.121890 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:07.121801 2578 generic.go:356] "Generic (PLEG): container finished" podID="e38ece80-57ff-4bbf-a9fe-15721b60478c" containerID="dda45e42518447a38d2037bbe095c09c9329823a5fdb01baed99e0f3e9c093c8" exitCode=0 Jul 02 13:16:07.121890 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:07.121874 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-5qxxr" event={"ID":"e38ece80-57ff-4bbf-a9fe-15721b60478c","Type":"ContainerDied","Data":"dda45e42518447a38d2037bbe095c09c9329823a5fdb01baed99e0f3e9c093c8"} Jul 02 13:16:07.122380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:07.122277 2578 scope.go:117] "RemoveContainer" containerID="dda45e42518447a38d2037bbe095c09c9329823a5fdb01baed99e0f3e9c093c8" Jul 02 13:16:07.123839 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:07.123817 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmc96" event={"ID":"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4","Type":"ContainerStarted","Data":"ea534727946eacf207a17bdafcd533dd3dff9f459cc0b91a12710d3b4298a003"} Jul 02 13:16:07.123898 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:07.123862 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmc96" event={"ID":"ac5d1704-1da6-48e2-b5fa-edecec3a5ee4","Type":"ContainerStarted","Data":"00a30320553246eefffa02b23058b51c38299dd0128ff808ef596d894627f40c"} Jul 02 13:16:07.164416 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:07.164361 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nmc96" podStartSLOduration=130.051192165 podStartE2EDuration="2m11.16434119s" totalImagesPullingTime="1.113149025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:07.164037433 +0000 UTC m=+132.008751135" watchObservedRunningTime="2026-07-02 13:16:07.16434119 +0000 UTC m=+132.009054870" Jul 02 13:16:08.129487 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:08.129442 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-5qxxr" event={"ID":"e38ece80-57ff-4bbf-a9fe-15721b60478c","Type":"ContainerStarted","Data":"50d6d2981ad8d235cf4961f61787af2d4d51b6e288d857af2a02ec7793a10aba"} Jul 02 13:16:08.252000 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:08.251967 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-87lm7_c5f79ccc-8051-4f78-b824-b1c8c6652212/serve-healthcheck-canary/0.log" Jul 02 13:16:08.648854 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:08.648815 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-9ddc5ff76-kq6rw_966752ab-e754-4b20-8e44-d0ef0947debf/cluster-samples-operator/0.log" Jul 02 13:16:08.845974 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:16:08.845941 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-9ddc5ff76-kq6rw_966752ab-e754-4b20-8e44-d0ef0947debf/cluster-samples-operator-watch/0.log" Jul 02 13:18:55.544146 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:18:55.544118 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:18:55.552807 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:18:55.552781 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:18:55.554677 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:18:55.554656 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 02 13:20:14.067091 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.067010 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-5zxnj"] Jul 02 13:20:14.070124 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.070107 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-5zxnj" Jul 02 13:20:14.072947 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.072907 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-pp7s2\"" Jul 02 13:20:14.073120 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.073104 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 02 13:20:14.074037 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.074019 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 02 13:20:14.074136 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.074114 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 02 13:20:14.084730 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.084707 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-5zxnj"] Jul 02 13:20:14.212249 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.212198 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4skkr\" (UniqueName: \"kubernetes.io/projected/415e88c8-d5c2-445a-9ebe-8befb5520084-kube-api-access-4skkr\") pod \"seaweedfs-64568bcd49-5zxnj\" (UID: \"415e88c8-d5c2-445a-9ebe-8befb5520084\") " pod="kserve/seaweedfs-64568bcd49-5zxnj" Jul 02 13:20:14.212249 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.212245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/415e88c8-d5c2-445a-9ebe-8befb5520084-data\") pod \"seaweedfs-64568bcd49-5zxnj\" (UID: \"415e88c8-d5c2-445a-9ebe-8befb5520084\") " pod="kserve/seaweedfs-64568bcd49-5zxnj" Jul 02 13:20:14.312987 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.312891 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4skkr\" (UniqueName: \"kubernetes.io/projected/415e88c8-d5c2-445a-9ebe-8befb5520084-kube-api-access-4skkr\") pod \"seaweedfs-64568bcd49-5zxnj\" (UID: \"415e88c8-d5c2-445a-9ebe-8befb5520084\") " pod="kserve/seaweedfs-64568bcd49-5zxnj" Jul 02 13:20:14.312987 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.312949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/415e88c8-d5c2-445a-9ebe-8befb5520084-data\") pod \"seaweedfs-64568bcd49-5zxnj\" (UID: \"415e88c8-d5c2-445a-9ebe-8befb5520084\") " pod="kserve/seaweedfs-64568bcd49-5zxnj" Jul 02 13:20:14.313292 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.313276 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/415e88c8-d5c2-445a-9ebe-8befb5520084-data\") pod \"seaweedfs-64568bcd49-5zxnj\" (UID: \"415e88c8-d5c2-445a-9ebe-8befb5520084\") " pod="kserve/seaweedfs-64568bcd49-5zxnj" Jul 02 13:20:14.323122 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.323069 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4skkr\" (UniqueName: \"kubernetes.io/projected/415e88c8-d5c2-445a-9ebe-8befb5520084-kube-api-access-4skkr\") pod \"seaweedfs-64568bcd49-5zxnj\" (UID: \"415e88c8-d5c2-445a-9ebe-8befb5520084\") " pod="kserve/seaweedfs-64568bcd49-5zxnj" Jul 02 13:20:14.379361 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.379329 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-5zxnj" Jul 02 13:20:14.510981 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.510952 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-5zxnj"] Jul 02 13:20:14.514543 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:20:14.514505 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415e88c8_d5c2_445a_9ebe_8befb5520084.slice/crio-7608ad4ad11eb770b40961cd2ea3d3d0289abe70d05db6447b0a9c3b2e1383c6 WatchSource:0}: Error finding container 7608ad4ad11eb770b40961cd2ea3d3d0289abe70d05db6447b0a9c3b2e1383c6: Status 404 returned error can't find the container with id 7608ad4ad11eb770b40961cd2ea3d3d0289abe70d05db6447b0a9c3b2e1383c6 Jul 02 13:20:14.516283 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.516265 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 13:20:14.980812 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:14.980772 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-5zxnj" event={"ID":"415e88c8-d5c2-445a-9ebe-8befb5520084","Type":"ContainerStarted","Data":"7608ad4ad11eb770b40961cd2ea3d3d0289abe70d05db6447b0a9c3b2e1383c6"} Jul 02 13:20:20.002188 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:20.002144 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-5zxnj" event={"ID":"415e88c8-d5c2-445a-9ebe-8befb5520084","Type":"ContainerStarted","Data":"2985468e03a314fbbbe8baa277525221e44bb92816065f4d709836b019a7dac2"} Jul 02 13:20:20.002626 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:20.002249 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-5zxnj" Jul 02 13:20:20.020661 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:20.020608 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-5zxnj" podStartSLOduration=1.416418426 podStartE2EDuration="6.020595078s" totalImagesPullingTime="4.604176652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:20:20.01940108 +0000 UTC m=+384.864114778" watchObservedRunningTime="2026-07-02 13:20:20.020595078 +0000 UTC m=+384.865308756" Jul 02 13:20:26.008390 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:20:26.008359 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-5zxnj" Jul 02 13:21:14.293676 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:14.293641 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz"] Jul 02 13:21:14.296923 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:14.296895 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" Jul 02 13:21:14.300414 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:14.300393 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 02 13:21:14.300749 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:14.300733 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xdqdg\"" Jul 02 13:21:14.300827 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:14.300752 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 02 13:21:14.310062 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:14.310041 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz"] Jul 02 13:21:14.354782 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:14.354755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f24818c-b736-4ed4-a27d-cdf6438f2f2d-tmp\") pod \"cert-manager-operator-controller-manager-5b66478c58-drdcz\" (UID: \"6f24818c-b736-4ed4-a27d-cdf6438f2f2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" Jul 02 13:21:14.354939 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:14.354792 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2w9k\" (UniqueName: \"kubernetes.io/projected/6f24818c-b736-4ed4-a27d-cdf6438f2f2d-kube-api-access-x2w9k\") pod \"cert-manager-operator-controller-manager-5b66478c58-drdcz\" (UID: \"6f24818c-b736-4ed4-a27d-cdf6438f2f2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" Jul 02 13:21:14.456012 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:14.455981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f24818c-b736-4ed4-a27d-cdf6438f2f2d-tmp\") pod \"cert-manager-operator-controller-manager-5b66478c58-drdcz\" (UID: \"6f24818c-b736-4ed4-a27d-cdf6438f2f2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" Jul 02 13:21:14.456012 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:14.456016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2w9k\" (UniqueName: \"kubernetes.io/projected/6f24818c-b736-4ed4-a27d-cdf6438f2f2d-kube-api-access-x2w9k\") pod \"cert-manager-operator-controller-manager-5b66478c58-drdcz\" (UID: \"6f24818c-b736-4ed4-a27d-cdf6438f2f2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" Jul 02 13:21:14.456402 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:14.456379 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f24818c-b736-4ed4-a27d-cdf6438f2f2d-tmp\") pod \"cert-manager-operator-controller-manager-5b66478c58-drdcz\" (UID: \"6f24818c-b736-4ed4-a27d-cdf6438f2f2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" Jul 02 13:21:14.465719 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:14.465693 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2w9k\" (UniqueName: \"kubernetes.io/projected/6f24818c-b736-4ed4-a27d-cdf6438f2f2d-kube-api-access-x2w9k\") pod \"cert-manager-operator-controller-manager-5b66478c58-drdcz\" (UID: \"6f24818c-b736-4ed4-a27d-cdf6438f2f2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" Jul 02 13:21:14.605758 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:14.605724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" Jul 02 13:21:14.744497 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:14.744463 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz"] Jul 02 13:21:14.748351 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:21:14.748328 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f24818c_b736_4ed4_a27d_cdf6438f2f2d.slice/crio-898b336e6e4e98f6be73a8d56416da1d1b181ac1ec4c695ddb732f20e91c978a WatchSource:0}: Error finding container 898b336e6e4e98f6be73a8d56416da1d1b181ac1ec4c695ddb732f20e91c978a: Status 404 returned error can't find the container with id 898b336e6e4e98f6be73a8d56416da1d1b181ac1ec4c695ddb732f20e91c978a Jul 02 13:21:15.197637 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:15.197599 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" event={"ID":"6f24818c-b736-4ed4-a27d-cdf6438f2f2d","Type":"ContainerStarted","Data":"898b336e6e4e98f6be73a8d56416da1d1b181ac1ec4c695ddb732f20e91c978a"} Jul 02 13:21:18.210768 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:18.210732 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" event={"ID":"6f24818c-b736-4ed4-a27d-cdf6438f2f2d","Type":"ContainerStarted","Data":"7b97ef8fd45b8592a6123225715ec47f2b165133d007a5b4deb7a5a647300699"} Jul 02 13:21:18.246182 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:18.246127 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" podStartSLOduration=1.595303903 podStartE2EDuration="4.246112257s" totalImagesPullingTime="2.650808354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:21:18.243830332 +0000 UTC m=+443.088544009" watchObservedRunningTime="2026-07-02 13:21:18.246112257 +0000 UTC m=+443.090825935" Jul 02 13:21:35.454622 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:35.454590 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q9cvl"] Jul 02 13:21:35.462045 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:35.462026 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q9cvl" Jul 02 13:21:35.465392 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:35.465371 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-vmmdl\"" Jul 02 13:21:35.466086 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:35.466068 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 02 13:21:35.466567 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:35.466550 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 02 13:21:35.470060 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:35.470038 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q9cvl"] Jul 02 13:21:35.518126 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:35.518099 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0082f696-05fe-49e4-b13c-ed5b85b1958d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q9cvl\" (UID: \"0082f696-05fe-49e4-b13c-ed5b85b1958d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q9cvl" Jul 02 13:21:35.518219 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:35.518140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsnvm\" (UniqueName: \"kubernetes.io/projected/0082f696-05fe-49e4-b13c-ed5b85b1958d-kube-api-access-gsnvm\") pod \"openshift-lws-operator-fd8ccff4c-q9cvl\" (UID: \"0082f696-05fe-49e4-b13c-ed5b85b1958d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q9cvl" Jul 02 13:21:35.619344 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:35.619318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0082f696-05fe-49e4-b13c-ed5b85b1958d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q9cvl\" (UID: \"0082f696-05fe-49e4-b13c-ed5b85b1958d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q9cvl" Jul 02 13:21:35.619438 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:35.619360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsnvm\" (UniqueName: \"kubernetes.io/projected/0082f696-05fe-49e4-b13c-ed5b85b1958d-kube-api-access-gsnvm\") pod \"openshift-lws-operator-fd8ccff4c-q9cvl\" (UID: \"0082f696-05fe-49e4-b13c-ed5b85b1958d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q9cvl" Jul 02 13:21:35.619759 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:35.619740 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0082f696-05fe-49e4-b13c-ed5b85b1958d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q9cvl\" (UID: \"0082f696-05fe-49e4-b13c-ed5b85b1958d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q9cvl" Jul 02 13:21:35.629677 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:35.629650 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsnvm\" (UniqueName: \"kubernetes.io/projected/0082f696-05fe-49e4-b13c-ed5b85b1958d-kube-api-access-gsnvm\") pod \"openshift-lws-operator-fd8ccff4c-q9cvl\" (UID: \"0082f696-05fe-49e4-b13c-ed5b85b1958d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q9cvl" Jul 02 13:21:35.771328 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:35.771262 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q9cvl" Jul 02 13:21:35.899004 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:35.898974 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q9cvl"] Jul 02 13:21:35.901865 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:21:35.901839 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0082f696_05fe_49e4_b13c_ed5b85b1958d.slice/crio-5adabb335bad7a6029968ea1a184605e7a081b58fe5380f602eaaa9c6a99dedd WatchSource:0}: Error finding container 5adabb335bad7a6029968ea1a184605e7a081b58fe5380f602eaaa9c6a99dedd: Status 404 returned error can't find the container with id 5adabb335bad7a6029968ea1a184605e7a081b58fe5380f602eaaa9c6a99dedd Jul 02 13:21:36.283893 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:36.283858 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q9cvl" event={"ID":"0082f696-05fe-49e4-b13c-ed5b85b1958d","Type":"ContainerStarted","Data":"5adabb335bad7a6029968ea1a184605e7a081b58fe5380f602eaaa9c6a99dedd"} Jul 02 13:21:39.302300 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:39.302217 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q9cvl" event={"ID":"0082f696-05fe-49e4-b13c-ed5b85b1958d","Type":"ContainerStarted","Data":"b9dfa907c226e41ef8a5534375318cd5a1aca15e75006fe897976be9e6504cbc"} Jul 02 13:21:39.320506 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:39.320458 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q9cvl" podStartSLOduration=1.219648433 podStartE2EDuration="4.320439918s" totalImagesPullingTime="3.100791485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:21:39.320160932 +0000 UTC m=+464.164874814" watchObservedRunningTime="2026-07-02 13:21:39.320439918 +0000 UTC m=+464.165153597" Jul 02 13:21:54.329879 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.329842 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz"] Jul 02 13:21:54.333544 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.333522 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.336600 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.336578 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 02 13:21:54.336848 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.336831 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 02 13:21:54.337201 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.337184 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-sng42\"" Jul 02 13:21:54.337957 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.337940 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 02 13:21:54.344866 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.344841 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz"] Jul 02 13:21:54.457693 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.457663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.457844 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.457698 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.457844 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.457725 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vjm8\" (UniqueName: \"kubernetes.io/projected/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-kube-api-access-2vjm8\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.457844 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.457793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.457844 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.457841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.458086 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.457877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.458086 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.457910 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.458086 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.457967 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.458086 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.457991 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.558807 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.558768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.558807 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.558810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.559052 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.558832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vjm8\" (UniqueName: \"kubernetes.io/projected/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-kube-api-access-2vjm8\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.559052 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.558878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.559052 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.558929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.559052 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.558967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.559052 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.558998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.559052 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.559027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.559052 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.559051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.559420 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.559326 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.559475 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.559454 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.559617 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.559577 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.559678 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.559645 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.559734 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.559688 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.561126 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.561103 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.561390 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.561370 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.567368 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.567342 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.567486 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.567470 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vjm8\" (UniqueName: \"kubernetes.io/projected/f2e2e7b8-5c47-41c9-8950-6bb74bb14be4-kube-api-access-2vjm8\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbgnz\" (UID: \"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.646716 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.646637 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:54.780421 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:54.780397 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz"] Jul 02 13:21:54.783041 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:21:54.783012 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e2e7b8_5c47_41c9_8950_6bb74bb14be4.slice/crio-86528c5e4243d9f82a97846f3fae6a45129539ca9af08b75f6e8c987fd247458 WatchSource:0}: Error finding container 86528c5e4243d9f82a97846f3fae6a45129539ca9af08b75f6e8c987fd247458: Status 404 returned error can't find the container with id 86528c5e4243d9f82a97846f3fae6a45129539ca9af08b75f6e8c987fd247458 Jul 02 13:21:55.367377 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:55.367327 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" event={"ID":"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4","Type":"ContainerStarted","Data":"86528c5e4243d9f82a97846f3fae6a45129539ca9af08b75f6e8c987fd247458"} Jul 02 13:21:57.326185 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:57.326143 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 02 13:21:57.326448 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:57.326221 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 02 13:21:57.326448 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:57.326248 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 02 13:21:58.380969 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:58.380909 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" event={"ID":"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4","Type":"ContainerStarted","Data":"1b2b3d965adbc48211a3cd6e320055cc32b13dbcf2b19694dbfd48e59c02c995"} Jul 02 13:21:58.402380 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:58.402333 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" podStartSLOduration=1.861438228 podStartE2EDuration="4.402317436s" totalImagesPullingTime="2.540879208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:21:58.401278424 +0000 UTC m=+483.245992102" watchObservedRunningTime="2026-07-02 13:21:58.402317436 +0000 UTC m=+483.247031113" Jul 02 13:21:58.647535 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:58.647436 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:58.651752 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:58.651727 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:59.095140 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.095095 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-676677444b-jz7rv"] Jul 02 13:21:59.098324 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.098300 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:21:59.101129 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.101107 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 02 13:21:59.101239 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.101191 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5bgv5\"" Jul 02 13:21:59.101303 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.101244 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 02 13:21:59.101456 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.101443 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 02 13:21:59.107664 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.107644 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-676677444b-jz7rv"] Jul 02 13:21:59.199335 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.199297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cca6b6ab-2b30-432c-8e4f-f9fad835f301-metrics-cert\") pod \"lws-controller-manager-676677444b-jz7rv\" (UID: \"cca6b6ab-2b30-432c-8e4f-f9fad835f301\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:21:59.199538 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.199373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cca6b6ab-2b30-432c-8e4f-f9fad835f301-manager-config\") pod \"lws-controller-manager-676677444b-jz7rv\" (UID: \"cca6b6ab-2b30-432c-8e4f-f9fad835f301\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:21:59.199538 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.199408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sb8x\" (UniqueName: \"kubernetes.io/projected/cca6b6ab-2b30-432c-8e4f-f9fad835f301-kube-api-access-4sb8x\") pod \"lws-controller-manager-676677444b-jz7rv\" (UID: \"cca6b6ab-2b30-432c-8e4f-f9fad835f301\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:21:59.199538 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.199447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cca6b6ab-2b30-432c-8e4f-f9fad835f301-cert\") pod \"lws-controller-manager-676677444b-jz7rv\" (UID: \"cca6b6ab-2b30-432c-8e4f-f9fad835f301\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:21:59.300179 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.300141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sb8x\" (UniqueName: \"kubernetes.io/projected/cca6b6ab-2b30-432c-8e4f-f9fad835f301-kube-api-access-4sb8x\") pod \"lws-controller-manager-676677444b-jz7rv\" (UID: \"cca6b6ab-2b30-432c-8e4f-f9fad835f301\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:21:59.300363 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.300190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cca6b6ab-2b30-432c-8e4f-f9fad835f301-cert\") pod \"lws-controller-manager-676677444b-jz7rv\" (UID: \"cca6b6ab-2b30-432c-8e4f-f9fad835f301\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:21:59.300363 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.300216 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cca6b6ab-2b30-432c-8e4f-f9fad835f301-metrics-cert\") pod \"lws-controller-manager-676677444b-jz7rv\" (UID: \"cca6b6ab-2b30-432c-8e4f-f9fad835f301\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:21:59.300363 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.300263 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cca6b6ab-2b30-432c-8e4f-f9fad835f301-manager-config\") pod \"lws-controller-manager-676677444b-jz7rv\" (UID: \"cca6b6ab-2b30-432c-8e4f-f9fad835f301\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:21:59.300864 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.300843 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cca6b6ab-2b30-432c-8e4f-f9fad835f301-manager-config\") pod \"lws-controller-manager-676677444b-jz7rv\" (UID: \"cca6b6ab-2b30-432c-8e4f-f9fad835f301\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:21:59.302656 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.302634 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cca6b6ab-2b30-432c-8e4f-f9fad835f301-cert\") pod \"lws-controller-manager-676677444b-jz7rv\" (UID: \"cca6b6ab-2b30-432c-8e4f-f9fad835f301\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:21:59.302741 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.302674 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cca6b6ab-2b30-432c-8e4f-f9fad835f301-metrics-cert\") pod \"lws-controller-manager-676677444b-jz7rv\" (UID: \"cca6b6ab-2b30-432c-8e4f-f9fad835f301\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:21:59.313147 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.313124 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sb8x\" (UniqueName: \"kubernetes.io/projected/cca6b6ab-2b30-432c-8e4f-f9fad835f301-kube-api-access-4sb8x\") pod \"lws-controller-manager-676677444b-jz7rv\" (UID: \"cca6b6ab-2b30-432c-8e4f-f9fad835f301\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:21:59.385102 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.385035 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:59.386067 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.386052 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:21:59.407991 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.407967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:21:59.551736 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:21:59.551684 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-676677444b-jz7rv"] Jul 02 13:21:59.554342 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:21:59.554317 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca6b6ab_2b30_432c_8e4f_f9fad835f301.slice/crio-ff94b09b17ae9a887add9aa3c220a9dd3d0f78d6527fa9f6f1ae9ca57be7fe89 WatchSource:0}: Error finding container ff94b09b17ae9a887add9aa3c220a9dd3d0f78d6527fa9f6f1ae9ca57be7fe89: Status 404 returned error can't find the container with id ff94b09b17ae9a887add9aa3c220a9dd3d0f78d6527fa9f6f1ae9ca57be7fe89 Jul 02 13:22:00.390229 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:00.390193 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" event={"ID":"cca6b6ab-2b30-432c-8e4f-f9fad835f301","Type":"ContainerStarted","Data":"ff94b09b17ae9a887add9aa3c220a9dd3d0f78d6527fa9f6f1ae9ca57be7fe89"} Jul 02 13:22:02.408422 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:02.408383 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" event={"ID":"cca6b6ab-2b30-432c-8e4f-f9fad835f301","Type":"ContainerStarted","Data":"80ae1b36ef9e703194e51004cf3fd36980227599603329c00ad3a2aeb45d1904"} Jul 02 13:22:02.408863 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:02.408510 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:22:02.426679 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:02.426624 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" podStartSLOduration=1.118281825 podStartE2EDuration="3.426608879s" totalImagesPullingTime="2.308327054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:22:02.425673207 +0000 UTC m=+487.270386903" watchObservedRunningTime="2026-07-02 13:22:02.426608879 +0000 UTC m=+487.271322607" Jul 02 13:22:13.414907 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:13.414875 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-676677444b-jz7rv" Jul 02 13:22:29.897043 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:29.897011 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr"] Jul 02 13:22:29.902442 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:29.902418 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr" Jul 02 13:22:29.905104 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:29.905075 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tzvvp\"" Jul 02 13:22:29.905245 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:29.905193 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 02 13:22:29.906250 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:29.906234 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 02 13:22:29.906316 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:29.906247 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 02 13:22:29.906496 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:29.906478 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 02 13:22:29.913577 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:29.913555 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr"] Jul 02 13:22:29.962958 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:29.962898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73fb5ed2-6f62-452c-8f2e-9a1d40a4411d-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-bn8mr\" (UID: \"73fb5ed2-6f62-452c-8f2e-9a1d40a4411d\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr" Jul 02 13:22:29.963149 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:29.962992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/73fb5ed2-6f62-452c-8f2e-9a1d40a4411d-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-bn8mr\" (UID: \"73fb5ed2-6f62-452c-8f2e-9a1d40a4411d\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr" Jul 02 13:22:29.963149 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:29.963025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdt6r\" (UniqueName: \"kubernetes.io/projected/73fb5ed2-6f62-452c-8f2e-9a1d40a4411d-kube-api-access-hdt6r\") pod \"kuadrant-console-plugin-85fd4b8b9d-bn8mr\" (UID: \"73fb5ed2-6f62-452c-8f2e-9a1d40a4411d\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr" Jul 02 13:22:30.064169 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:30.064132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73fb5ed2-6f62-452c-8f2e-9a1d40a4411d-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-bn8mr\" (UID: \"73fb5ed2-6f62-452c-8f2e-9a1d40a4411d\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr" Jul 02 13:22:30.064351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:30.064182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/73fb5ed2-6f62-452c-8f2e-9a1d40a4411d-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-bn8mr\" (UID: \"73fb5ed2-6f62-452c-8f2e-9a1d40a4411d\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr" Jul 02 13:22:30.064351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:30.064207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdt6r\" (UniqueName: \"kubernetes.io/projected/73fb5ed2-6f62-452c-8f2e-9a1d40a4411d-kube-api-access-hdt6r\") pod \"kuadrant-console-plugin-85fd4b8b9d-bn8mr\" (UID: \"73fb5ed2-6f62-452c-8f2e-9a1d40a4411d\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr" Jul 02 13:22:30.064962 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:30.064906 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73fb5ed2-6f62-452c-8f2e-9a1d40a4411d-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-bn8mr\" (UID: \"73fb5ed2-6f62-452c-8f2e-9a1d40a4411d\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr" Jul 02 13:22:30.066873 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:30.066851 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/73fb5ed2-6f62-452c-8f2e-9a1d40a4411d-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-bn8mr\" (UID: \"73fb5ed2-6f62-452c-8f2e-9a1d40a4411d\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr" Jul 02 13:22:30.078216 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:30.078185 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdt6r\" (UniqueName: \"kubernetes.io/projected/73fb5ed2-6f62-452c-8f2e-9a1d40a4411d-kube-api-access-hdt6r\") pod \"kuadrant-console-plugin-85fd4b8b9d-bn8mr\" (UID: \"73fb5ed2-6f62-452c-8f2e-9a1d40a4411d\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr" Jul 02 13:22:30.213103 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:30.213011 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr" Jul 02 13:22:30.346059 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:30.346026 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr"] Jul 02 13:22:30.349241 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:22:30.349212 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73fb5ed2_6f62_452c_8f2e_9a1d40a4411d.slice/crio-826393b3b7bd4c184798a2ebdb96cf4d7d5864276d8bf751da57cc51986207c9 WatchSource:0}: Error finding container 826393b3b7bd4c184798a2ebdb96cf4d7d5864276d8bf751da57cc51986207c9: Status 404 returned error can't find the container with id 826393b3b7bd4c184798a2ebdb96cf4d7d5864276d8bf751da57cc51986207c9 Jul 02 13:22:30.529590 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:30.529498 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr" event={"ID":"73fb5ed2-6f62-452c-8f2e-9a1d40a4411d","Type":"ContainerStarted","Data":"826393b3b7bd4c184798a2ebdb96cf4d7d5864276d8bf751da57cc51986207c9"} Jul 02 13:22:35.557252 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:35.557167 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr" event={"ID":"73fb5ed2-6f62-452c-8f2e-9a1d40a4411d","Type":"ContainerStarted","Data":"aa8456dfbcdf03c8003d0d767166bcc52c944487dba01e7c0875dac7d50dbd21"} Jul 02 13:22:35.573361 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:22:35.573311 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr" podStartSLOduration=1.699576502 podStartE2EDuration="6.573294686s" totalImagesPullingTime="4.873718184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:22:35.573243666 +0000 UTC m=+520.417957344" watchObservedRunningTime="2026-07-02 13:22:35.573294686 +0000 UTC m=+520.418008365" Jul 02 13:23:13.241319 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:13.241233 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-6f84d"] Jul 02 13:23:13.260676 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:13.260640 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-6f84d"] Jul 02 13:23:13.260849 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:13.260754 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-6f84d" Jul 02 13:23:13.263685 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:13.263661 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 02 13:23:13.276188 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:13.276165 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-6f84d"] Jul 02 13:23:13.309634 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:13.309606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3cf77edd-bbc3-4292-a811-54328c9d837d-config-file\") pod \"limitador-limitador-69574b596d-6f84d\" (UID: \"3cf77edd-bbc3-4292-a811-54328c9d837d\") " pod="kuadrant-system/limitador-limitador-69574b596d-6f84d" Jul 02 13:23:13.309737 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:13.309655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9fhj\" (UniqueName: \"kubernetes.io/projected/3cf77edd-bbc3-4292-a811-54328c9d837d-kube-api-access-v9fhj\") pod \"limitador-limitador-69574b596d-6f84d\" (UID: \"3cf77edd-bbc3-4292-a811-54328c9d837d\") " pod="kuadrant-system/limitador-limitador-69574b596d-6f84d" Jul 02 13:23:13.411030 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:13.410999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9fhj\" (UniqueName: \"kubernetes.io/projected/3cf77edd-bbc3-4292-a811-54328c9d837d-kube-api-access-v9fhj\") pod \"limitador-limitador-69574b596d-6f84d\" (UID: \"3cf77edd-bbc3-4292-a811-54328c9d837d\") " pod="kuadrant-system/limitador-limitador-69574b596d-6f84d" Jul 02 13:23:13.411162 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:13.411082 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3cf77edd-bbc3-4292-a811-54328c9d837d-config-file\") pod \"limitador-limitador-69574b596d-6f84d\" (UID: \"3cf77edd-bbc3-4292-a811-54328c9d837d\") " pod="kuadrant-system/limitador-limitador-69574b596d-6f84d" Jul 02 13:23:13.411664 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:13.411646 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3cf77edd-bbc3-4292-a811-54328c9d837d-config-file\") pod \"limitador-limitador-69574b596d-6f84d\" (UID: \"3cf77edd-bbc3-4292-a811-54328c9d837d\") " pod="kuadrant-system/limitador-limitador-69574b596d-6f84d" Jul 02 13:23:13.421374 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:13.421348 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9fhj\" (UniqueName: \"kubernetes.io/projected/3cf77edd-bbc3-4292-a811-54328c9d837d-kube-api-access-v9fhj\") pod \"limitador-limitador-69574b596d-6f84d\" (UID: \"3cf77edd-bbc3-4292-a811-54328c9d837d\") " pod="kuadrant-system/limitador-limitador-69574b596d-6f84d" Jul 02 13:23:13.570861 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:13.570829 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-6f84d" Jul 02 13:23:13.704282 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:13.704221 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-6f84d"] Jul 02 13:23:13.706946 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:23:13.706901 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf77edd_bbc3_4292_a811_54328c9d837d.slice/crio-eaaff305dff385026cadb11366500b96b60868d2f7d976ff1878410cccf3efaa WatchSource:0}: Error finding container eaaff305dff385026cadb11366500b96b60868d2f7d976ff1878410cccf3efaa: Status 404 returned error can't find the container with id eaaff305dff385026cadb11366500b96b60868d2f7d976ff1878410cccf3efaa Jul 02 13:23:14.714614 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:14.714576 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-6f84d" event={"ID":"3cf77edd-bbc3-4292-a811-54328c9d837d","Type":"ContainerStarted","Data":"eaaff305dff385026cadb11366500b96b60868d2f7d976ff1878410cccf3efaa"} Jul 02 13:23:18.735443 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:18.735407 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-6f84d" event={"ID":"3cf77edd-bbc3-4292-a811-54328c9d837d","Type":"ContainerStarted","Data":"3df43d6d59575138fdde0d0e85bba0125e4cf39c67b08be063375d53dc71d059"} Jul 02 13:23:18.735910 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:18.735534 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-69574b596d-6f84d" Jul 02 13:23:18.754945 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:18.754884 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-69574b596d-6f84d" podStartSLOduration=1.306521016 podStartE2EDuration="5.754871126s" totalImagesPullingTime="4.44835011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:23:18.752476317 +0000 UTC m=+563.597189997" watchObservedRunningTime="2026-07-02 13:23:18.754871126 +0000 UTC m=+563.599584805" Jul 02 13:23:29.741263 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:29.741234 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-69574b596d-6f84d" Jul 02 13:23:55.571704 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:55.571668 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:23:55.577150 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:23:55.577121 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:26:52.633647 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:52.633613 2578 generic.go:356] "Generic (PLEG): container finished" podID="f2e2e7b8-5c47-41c9-8950-6bb74bb14be4" containerID="1b2b3d965adbc48211a3cd6e320055cc32b13dbcf2b19694dbfd48e59c02c995" exitCode=0 Jul 02 13:26:52.633647 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:52.633650 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" event={"ID":"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4","Type":"ContainerDied","Data":"1b2b3d965adbc48211a3cd6e320055cc32b13dbcf2b19694dbfd48e59c02c995"} Jul 02 13:26:52.634127 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:52.633991 2578 scope.go:117] "RemoveContainer" containerID="1b2b3d965adbc48211a3cd6e320055cc32b13dbcf2b19694dbfd48e59c02c995" Jul 02 13:26:52.634884 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:52.634849 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 02 13:26:52.635014 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:52.634935 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 02 13:26:52.635014 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:52.634973 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 02 13:26:53.642285 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:53.642242 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" event={"ID":"f2e2e7b8-5c47-41c9-8950-6bb74bb14be4","Type":"ContainerStarted","Data":"72da0c46683ec03f5d1d9d7c91dafadfd4e543181b59155e89145429aad1a58f"} Jul 02 13:26:53.647236 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:53.647212 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:26:53.647511 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:53.647355 2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 02 13:26:53.647687 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:53.647658 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" podUID="f2e2e7b8-5c47-41c9-8950-6bb74bb14be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 02 13:26:54.649392 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:54.649351 2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 02 13:26:54.649870 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:54.649412 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" podUID="f2e2e7b8-5c47-41c9-8950-6bb74bb14be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 02 13:26:55.647761 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:55.647720 2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 02 13:26:55.647985 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:55.647790 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" podUID="f2e2e7b8-5c47-41c9-8950-6bb74bb14be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 02 13:26:56.647429 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:56.647391 2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 02 13:26:56.648032 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:56.648000 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" podUID="f2e2e7b8-5c47-41c9-8950-6bb74bb14be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 02 13:26:56.993279 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:56.993201 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx"] Jul 02 13:26:56.995372 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:56.995355 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:56.998130 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:56.998110 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\"" Jul 02 13:26:56.998228 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:56.998155 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 02 13:26:56.999164 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:56.999121 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 02 13:26:56.999503 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:56.999487 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\"" Jul 02 13:26:57.009218 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.009197 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx"] Jul 02 13:26:57.054358 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.054322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.054358 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.054358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.054559 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.054392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.054559 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.054515 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51bcd2d1-7f66-49ff-acc5-040f638b8d03-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.054627 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.054570 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfjdw\" (UniqueName: \"kubernetes.io/projected/51bcd2d1-7f66-49ff-acc5-040f638b8d03-kube-api-access-nfjdw\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.054660 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.054624 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.155549 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.155510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.155737 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.155581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.155737 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.155606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.155737 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.155637 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.155737 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.155698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51bcd2d1-7f66-49ff-acc5-040f638b8d03-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.155979 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.155737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfjdw\" (UniqueName: \"kubernetes.io/projected/51bcd2d1-7f66-49ff-acc5-040f638b8d03-kube-api-access-nfjdw\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.155979 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.155950 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.156082 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.156000 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.156214 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.156191 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.158195 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.158170 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.158829 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.158808 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51bcd2d1-7f66-49ff-acc5-040f638b8d03-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.166385 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.166366 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfjdw\" (UniqueName: \"kubernetes.io/projected/51bcd2d1-7f66-49ff-acc5-040f638b8d03-kube-api-access-nfjdw\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.305997 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.305874 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:26:57.483540 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.483516 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx"] Jul 02 13:26:57.486492 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:26:57.486472 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51bcd2d1_7f66_49ff_acc5_040f638b8d03.slice/crio-fb0fb60a7c2386919f8625d3330256c49cc306cb82f91e6642d5064741b8287c WatchSource:0}: Error finding container fb0fb60a7c2386919f8625d3330256c49cc306cb82f91e6642d5064741b8287c: Status 404 returned error can't find the container with id fb0fb60a7c2386919f8625d3330256c49cc306cb82f91e6642d5064741b8287c Jul 02 13:26:57.488415 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.488399 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 13:26:57.647354 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.647311 2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 02 13:26:57.647548 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.647383 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" podUID="f2e2e7b8-5c47-41c9-8950-6bb74bb14be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 02 13:26:57.662825 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:57.662796 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" event={"ID":"51bcd2d1-7f66-49ff-acc5-040f638b8d03","Type":"ContainerStarted","Data":"fb0fb60a7c2386919f8625d3330256c49cc306cb82f91e6642d5064741b8287c"} Jul 02 13:26:58.876559 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:58.876517 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:26:58.877194 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:58.877168 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:26:58.878285 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:58.878255 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz" Jul 02 13:26:59.674847 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:59.674811 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" event={"ID":"51bcd2d1-7f66-49ff-acc5-040f638b8d03","Type":"ContainerStarted","Data":"144c18184ede97553e76a9e6141eaefa32e109c8f265abf586162bdb7b2a9544"} Jul 02 13:26:59.697001 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:26:59.696956 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" podStartSLOduration=2.436603599 podStartE2EDuration="3.696942284s" totalImagesPullingTime="1.260338685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:26:59.695997348 +0000 UTC m=+784.540711025" watchObservedRunningTime="2026-07-02 13:26:59.696942284 +0000 UTC m=+784.541655960" Jul 02 13:27:00.306931 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:00.306896 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:27:07.306449 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:07.306417 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:27:07.318001 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:07.317980 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:27:07.718874 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:07.718846 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:27:09.901284 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:09.901250 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx"] Jul 02 13:27:24.920763 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:24.920724 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" podUID="51bcd2d1-7f66-49ff-acc5-040f638b8d03" containerName="main" containerID="cri-o://144c18184ede97553e76a9e6141eaefa32e109c8f265abf586162bdb7b2a9544" gracePeriod=45 Jul 02 13:27:25.174063 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.174005 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:27:25.274101 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.274070 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-home\") pod \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " Jul 02 13:27:25.274296 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.274125 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51bcd2d1-7f66-49ff-acc5-040f638b8d03-tls-certs\") pod \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " Jul 02 13:27:25.274296 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.274168 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-tmp-dir\") pod \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " Jul 02 13:27:25.274296 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.274247 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfjdw\" (UniqueName: \"kubernetes.io/projected/51bcd2d1-7f66-49ff-acc5-040f638b8d03-kube-api-access-nfjdw\") pod \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " Jul 02 13:27:25.274458 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.274308 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-model-cache\") pod \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " Jul 02 13:27:25.274458 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.274337 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-dshm\") pod \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\" (UID: \"51bcd2d1-7f66-49ff-acc5-040f638b8d03\") " Jul 02 13:27:25.274458 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.274410 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-home" (OuterVolumeSpecName: "home") pod "51bcd2d1-7f66-49ff-acc5-040f638b8d03" (UID: "51bcd2d1-7f66-49ff-acc5-040f638b8d03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:27:25.274595 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.274551 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-home\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:27:25.274890 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.274851 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-model-cache" (OuterVolumeSpecName: "model-cache") pod "51bcd2d1-7f66-49ff-acc5-040f638b8d03" (UID: "51bcd2d1-7f66-49ff-acc5-040f638b8d03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:27:25.275013 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.274971 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51bcd2d1-7f66-49ff-acc5-040f638b8d03" (UID: "51bcd2d1-7f66-49ff-acc5-040f638b8d03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:27:25.276731 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.276704 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51bcd2d1-7f66-49ff-acc5-040f638b8d03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51bcd2d1-7f66-49ff-acc5-040f638b8d03" (UID: "51bcd2d1-7f66-49ff-acc5-040f638b8d03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 13:27:25.276963 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.276941 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51bcd2d1-7f66-49ff-acc5-040f638b8d03-kube-api-access-nfjdw" (OuterVolumeSpecName: "kube-api-access-nfjdw") pod "51bcd2d1-7f66-49ff-acc5-040f638b8d03" (UID: "51bcd2d1-7f66-49ff-acc5-040f638b8d03"). InnerVolumeSpecName "kube-api-access-nfjdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 13:27:25.277097 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.277026 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-dshm" (OuterVolumeSpecName: "dshm") pod "51bcd2d1-7f66-49ff-acc5-040f638b8d03" (UID: "51bcd2d1-7f66-49ff-acc5-040f638b8d03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:27:25.375160 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.375113 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-dshm\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:27:25.375160 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.375140 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51bcd2d1-7f66-49ff-acc5-040f638b8d03-tls-certs\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:27:25.375322 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.375188 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-tmp-dir\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:27:25.375322 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.375197 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfjdw\" (UniqueName: \"kubernetes.io/projected/51bcd2d1-7f66-49ff-acc5-040f638b8d03-kube-api-access-nfjdw\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:27:25.375322 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.375208 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51bcd2d1-7f66-49ff-acc5-040f638b8d03-model-cache\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:27:25.784601 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.784565 2578 generic.go:356] "Generic (PLEG): container finished" podID="51bcd2d1-7f66-49ff-acc5-040f638b8d03" containerID="144c18184ede97553e76a9e6141eaefa32e109c8f265abf586162bdb7b2a9544" exitCode=0 Jul 02 13:27:25.784751 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.784617 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" event={"ID":"51bcd2d1-7f66-49ff-acc5-040f638b8d03","Type":"ContainerDied","Data":"144c18184ede97553e76a9e6141eaefa32e109c8f265abf586162bdb7b2a9544"} Jul 02 13:27:25.784751 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.784647 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" Jul 02 13:27:25.784751 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.784669 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx" event={"ID":"51bcd2d1-7f66-49ff-acc5-040f638b8d03","Type":"ContainerDied","Data":"fb0fb60a7c2386919f8625d3330256c49cc306cb82f91e6642d5064741b8287c"} Jul 02 13:27:25.784751 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.784699 2578 scope.go:117] "RemoveContainer" containerID="144c18184ede97553e76a9e6141eaefa32e109c8f265abf586162bdb7b2a9544" Jul 02 13:27:25.795733 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.795672 2578 scope.go:117] "RemoveContainer" containerID="144c18184ede97553e76a9e6141eaefa32e109c8f265abf586162bdb7b2a9544" Jul 02 13:27:25.796061 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:27:25.796026 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144c18184ede97553e76a9e6141eaefa32e109c8f265abf586162bdb7b2a9544\": container with ID starting with 144c18184ede97553e76a9e6141eaefa32e109c8f265abf586162bdb7b2a9544 not found: ID does not exist" containerID="144c18184ede97553e76a9e6141eaefa32e109c8f265abf586162bdb7b2a9544" Jul 02 13:27:25.796166 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.796074 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144c18184ede97553e76a9e6141eaefa32e109c8f265abf586162bdb7b2a9544"} err="failed to get container status \"144c18184ede97553e76a9e6141eaefa32e109c8f265abf586162bdb7b2a9544\": rpc error: code = NotFound desc = could not find container \"144c18184ede97553e76a9e6141eaefa32e109c8f265abf586162bdb7b2a9544\": container with ID starting with 144c18184ede97553e76a9e6141eaefa32e109c8f265abf586162bdb7b2a9544 not found: ID does not exist" Jul 02 13:27:25.812097 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.812067 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx"] Jul 02 13:27:25.817315 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:25.817292 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx"] Jul 02 13:27:27.624303 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:27:27.624264 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51bcd2d1-7f66-49ff-acc5-040f638b8d03" path="/var/lib/kubelet/pods/51bcd2d1-7f66-49ff-acc5-040f638b8d03/volumes" Jul 02 13:28:55.599247 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:28:55.599170 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:28:55.602588 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:28:55.602567 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:29:22.322817 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.322780 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw"] Jul 02 13:29:22.323303 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.323156 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51bcd2d1-7f66-49ff-acc5-040f638b8d03" containerName="main" Jul 02 13:29:22.323303 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.323169 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="51bcd2d1-7f66-49ff-acc5-040f638b8d03" containerName="main" Jul 02 13:29:22.323303 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.323222 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="51bcd2d1-7f66-49ff-acc5-040f638b8d03" containerName="main" Jul 02 13:29:22.325971 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.325955 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:29:22.328802 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.328778 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 02 13:29:22.328953 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.328842 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 02 13:29:22.329847 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.329829 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\"" Jul 02 13:29:22.330123 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.330107 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-9xjgt\"" Jul 02 13:29:22.330209 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.330164 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\"" Jul 02 13:29:22.338578 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.338555 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw"] Jul 02 13:29:22.422005 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.421960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/25cd1f11-951d-4d4c-90bb-0be7f344f923-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw\" (UID: \"25cd1f11-951d-4d4c-90bb-0be7f344f923\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:29:22.422216 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.422047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs6tm\" (UniqueName: \"kubernetes.io/projected/25cd1f11-951d-4d4c-90bb-0be7f344f923-kube-api-access-rs6tm\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw\" (UID: \"25cd1f11-951d-4d4c-90bb-0be7f344f923\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:29:22.422216 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.422125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25cd1f11-951d-4d4c-90bb-0be7f344f923-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw\" (UID: \"25cd1f11-951d-4d4c-90bb-0be7f344f923\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:29:22.522564 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.522517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs6tm\" (UniqueName: \"kubernetes.io/projected/25cd1f11-951d-4d4c-90bb-0be7f344f923-kube-api-access-rs6tm\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw\" (UID: \"25cd1f11-951d-4d4c-90bb-0be7f344f923\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:29:22.522780 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.522575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25cd1f11-951d-4d4c-90bb-0be7f344f923-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw\" (UID: \"25cd1f11-951d-4d4c-90bb-0be7f344f923\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:29:22.522780 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.522677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/25cd1f11-951d-4d4c-90bb-0be7f344f923-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw\" (UID: \"25cd1f11-951d-4d4c-90bb-0be7f344f923\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:29:22.523049 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.523033 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/25cd1f11-951d-4d4c-90bb-0be7f344f923-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw\" (UID: \"25cd1f11-951d-4d4c-90bb-0be7f344f923\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:29:22.525180 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.525157 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25cd1f11-951d-4d4c-90bb-0be7f344f923-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw\" (UID: \"25cd1f11-951d-4d4c-90bb-0be7f344f923\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:29:22.532985 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.532959 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs6tm\" (UniqueName: \"kubernetes.io/projected/25cd1f11-951d-4d4c-90bb-0be7f344f923-kube-api-access-rs6tm\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw\" (UID: \"25cd1f11-951d-4d4c-90bb-0be7f344f923\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:29:22.635682 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.635594 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:29:22.773087 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:22.773056 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw"] Jul 02 13:29:23.263607 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:23.263566 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" event={"ID":"25cd1f11-951d-4d4c-90bb-0be7f344f923","Type":"ContainerStarted","Data":"60eb8368c9f3b2d7d11fb196b381b3643b4e7b554f4c8fe3d9c0894ca9437708"} Jul 02 13:29:25.273119 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:25.273084 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" event={"ID":"25cd1f11-951d-4d4c-90bb-0be7f344f923","Type":"ContainerStarted","Data":"7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57"} Jul 02 13:29:25.273491 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:25.273181 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:29:25.298798 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:25.298752 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" podStartSLOduration=1.8004822470000001 podStartE2EDuration="3.298737908s" totalImagesPullingTime="1.498255661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:29:25.297551497 +0000 UTC m=+930.142265187" watchObservedRunningTime="2026-07-02 13:29:25.298737908 +0000 UTC m=+930.143451586" Jul 02 13:29:56.279527 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:29:56.279497 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:31:19.487308 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.487272 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk"] Jul 02 13:31:19.490713 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.490692 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.493612 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.493592 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\"" Jul 02 13:31:19.502936 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.502893 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk"] Jul 02 13:31:19.579334 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.579301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.579501 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.579348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.579501 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.579449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.579501 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.579474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/106fb5e6-6779-444a-8d1c-0503a11a73ce-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.579613 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.579522 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.579613 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.579542 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk9s9\" (UniqueName: \"kubernetes.io/projected/106fb5e6-6779-444a-8d1c-0503a11a73ce-kube-api-access-lk9s9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.680460 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.680429 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.680460 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.680464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/106fb5e6-6779-444a-8d1c-0503a11a73ce-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.680723 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.680513 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.680723 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.680642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9s9\" (UniqueName: \"kubernetes.io/projected/106fb5e6-6779-444a-8d1c-0503a11a73ce-kube-api-access-lk9s9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.680723 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.680716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.680881 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.680807 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.680960 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.680941 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.681080 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.681058 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.681194 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.681178 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.682812 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.682789 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.683140 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.683120 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/106fb5e6-6779-444a-8d1c-0503a11a73ce-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.690324 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.690303 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9s9\" (UniqueName: \"kubernetes.io/projected/106fb5e6-6779-444a-8d1c-0503a11a73ce-kube-api-access-lk9s9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.800903 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.800792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:19.947308 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:19.947276 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk"] Jul 02 13:31:19.950641 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:31:19.950608 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106fb5e6_6779_444a_8d1c_0503a11a73ce.slice/crio-f5a5ec2ab8941e538e94684f703904a8474b5fb74f2a2c6973b98bb64cf47f40 WatchSource:0}: Error finding container f5a5ec2ab8941e538e94684f703904a8474b5fb74f2a2c6973b98bb64cf47f40: Status 404 returned error can't find the container with id f5a5ec2ab8941e538e94684f703904a8474b5fb74f2a2c6973b98bb64cf47f40 Jul 02 13:31:20.773925 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:20.773877 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" event={"ID":"106fb5e6-6779-444a-8d1c-0503a11a73ce","Type":"ContainerStarted","Data":"c5587c1388a8b5ae76c070c17d83de08ac936ccd45497cf9332a95f69a462985"} Jul 02 13:31:20.774286 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:20.773941 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" event={"ID":"106fb5e6-6779-444a-8d1c-0503a11a73ce","Type":"ContainerStarted","Data":"f5a5ec2ab8941e538e94684f703904a8474b5fb74f2a2c6973b98bb64cf47f40"} Jul 02 13:31:20.794823 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:20.794766 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" podStartSLOduration=1.794748234 podStartE2EDuration="1.794748234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:31:20.794343927 +0000 UTC m=+1045.639057632" watchObservedRunningTime="2026-07-02 13:31:20.794748234 +0000 UTC m=+1045.639461913" Jul 02 13:31:20.801142 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:20.801118 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:29.801905 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:29.801867 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:29.814379 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:29.814351 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:29.825362 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:29.825340 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:31:53.571792 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:31:53.571758 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk"] Jul 02 13:32:08.592557 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.592510 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" podUID="106fb5e6-6779-444a-8d1c-0503a11a73ce" containerName="main" containerID="cri-o://c5587c1388a8b5ae76c070c17d83de08ac936ccd45497cf9332a95f69a462985" gracePeriod=45 Jul 02 13:32:08.841054 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.841030 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:32:08.887928 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.887846 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk9s9\" (UniqueName: \"kubernetes.io/projected/106fb5e6-6779-444a-8d1c-0503a11a73ce-kube-api-access-lk9s9\") pod \"106fb5e6-6779-444a-8d1c-0503a11a73ce\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " Jul 02 13:32:08.887928 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.887901 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-tmp-dir\") pod \"106fb5e6-6779-444a-8d1c-0503a11a73ce\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " Jul 02 13:32:08.888109 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.887961 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-model-cache\") pod \"106fb5e6-6779-444a-8d1c-0503a11a73ce\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " Jul 02 13:32:08.888109 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.887981 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-dshm\") pod \"106fb5e6-6779-444a-8d1c-0503a11a73ce\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " Jul 02 13:32:08.888109 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.888008 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/106fb5e6-6779-444a-8d1c-0503a11a73ce-tls-certs\") pod \"106fb5e6-6779-444a-8d1c-0503a11a73ce\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " Jul 02 13:32:08.888270 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.888244 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "106fb5e6-6779-444a-8d1c-0503a11a73ce" (UID: "106fb5e6-6779-444a-8d1c-0503a11a73ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:32:08.888323 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.888272 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "106fb5e6-6779-444a-8d1c-0503a11a73ce" (UID: "106fb5e6-6779-444a-8d1c-0503a11a73ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:32:08.889997 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.889973 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106fb5e6-6779-444a-8d1c-0503a11a73ce-kube-api-access-lk9s9" (OuterVolumeSpecName: "kube-api-access-lk9s9") pod "106fb5e6-6779-444a-8d1c-0503a11a73ce" (UID: "106fb5e6-6779-444a-8d1c-0503a11a73ce"). InnerVolumeSpecName "kube-api-access-lk9s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 13:32:08.889997 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.889980 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-dshm" (OuterVolumeSpecName: "dshm") pod "106fb5e6-6779-444a-8d1c-0503a11a73ce" (UID: "106fb5e6-6779-444a-8d1c-0503a11a73ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:32:08.890101 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.890046 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/106fb5e6-6779-444a-8d1c-0503a11a73ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "106fb5e6-6779-444a-8d1c-0503a11a73ce" (UID: "106fb5e6-6779-444a-8d1c-0503a11a73ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 13:32:08.987513 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.987485 2578 generic.go:356] "Generic (PLEG): container finished" podID="106fb5e6-6779-444a-8d1c-0503a11a73ce" containerID="c5587c1388a8b5ae76c070c17d83de08ac936ccd45497cf9332a95f69a462985" exitCode=0 Jul 02 13:32:08.987655 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.987538 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" Jul 02 13:32:08.987655 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.987573 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" event={"ID":"106fb5e6-6779-444a-8d1c-0503a11a73ce","Type":"ContainerDied","Data":"c5587c1388a8b5ae76c070c17d83de08ac936ccd45497cf9332a95f69a462985"} Jul 02 13:32:08.987655 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.987624 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" event={"ID":"106fb5e6-6779-444a-8d1c-0503a11a73ce","Type":"ContainerDied","Data":"f5a5ec2ab8941e538e94684f703904a8474b5fb74f2a2c6973b98bb64cf47f40"} Jul 02 13:32:08.987655 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.987649 2578 scope.go:117] "RemoveContainer" containerID="c5587c1388a8b5ae76c070c17d83de08ac936ccd45497cf9332a95f69a462985" Jul 02 13:32:08.988410 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.988393 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-home\") pod \"106fb5e6-6779-444a-8d1c-0503a11a73ce\" (UID: \"106fb5e6-6779-444a-8d1c-0503a11a73ce\") " Jul 02 13:32:08.988668 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.988604 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk9s9\" (UniqueName: \"kubernetes.io/projected/106fb5e6-6779-444a-8d1c-0503a11a73ce-kube-api-access-lk9s9\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:32:08.988723 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.988673 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-tmp-dir\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:32:08.988723 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.988674 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-home" (OuterVolumeSpecName: "home") pod "106fb5e6-6779-444a-8d1c-0503a11a73ce" (UID: "106fb5e6-6779-444a-8d1c-0503a11a73ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:32:08.988723 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.988683 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-model-cache\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:32:08.988824 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.988726 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-dshm\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:32:08.988824 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.988741 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/106fb5e6-6779-444a-8d1c-0503a11a73ce-tls-certs\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:32:08.995733 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.995714 2578 scope.go:117] "RemoveContainer" containerID="c5587c1388a8b5ae76c070c17d83de08ac936ccd45497cf9332a95f69a462985" Jul 02 13:32:08.996006 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:32:08.995986 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5587c1388a8b5ae76c070c17d83de08ac936ccd45497cf9332a95f69a462985\": container with ID starting with c5587c1388a8b5ae76c070c17d83de08ac936ccd45497cf9332a95f69a462985 not found: ID does not exist" containerID="c5587c1388a8b5ae76c070c17d83de08ac936ccd45497cf9332a95f69a462985" Jul 02 13:32:08.996053 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:08.996015 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5587c1388a8b5ae76c070c17d83de08ac936ccd45497cf9332a95f69a462985"} err="failed to get container status \"c5587c1388a8b5ae76c070c17d83de08ac936ccd45497cf9332a95f69a462985\": rpc error: code = NotFound desc = could not find container \"c5587c1388a8b5ae76c070c17d83de08ac936ccd45497cf9332a95f69a462985\": container with ID starting with c5587c1388a8b5ae76c070c17d83de08ac936ccd45497cf9332a95f69a462985 not found: ID does not exist" Jul 02 13:32:09.089558 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:09.089532 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/106fb5e6-6779-444a-8d1c-0503a11a73ce-home\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:32:09.310265 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:09.310240 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk"] Jul 02 13:32:09.314068 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:09.314045 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk"] Jul 02 13:32:09.625694 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:09.625665 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="106fb5e6-6779-444a-8d1c-0503a11a73ce" path="/var/lib/kubelet/pods/106fb5e6-6779-444a-8d1c-0503a11a73ce/volumes" Jul 02 13:32:09.815744 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:09.815702 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk" podUID="106fb5e6-6779-444a-8d1c-0503a11a73ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 02 13:32:13.503069 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:13.503039 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw"] Jul 02 13:32:13.511948 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:32:13.511896 2578 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 02 13:32:13.511948 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 13:32:13.511948 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 13:32:13.511948 ip-10-0-131-243 kubenswrapper[2578]: > containerID="7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57" cmd=["/bin/sleep","15"] Jul 02 13:32:13.512254 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:13.511983 2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" message="" err=< Jul 02 13:32:13.512254 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 13:32:13.512254 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 13:32:13.512254 ip-10-0-131-243 kubenswrapper[2578]: > Jul 02 13:32:13.512254 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:32:13.512007 2578 kuberuntime_container.go:754] "PreStop hook failed" err=< Jul 02 13:32:13.512254 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 13:32:13.512254 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 13:32:13.512254 ip-10-0-131-243 kubenswrapper[2578]: > pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" podUID="25cd1f11-951d-4d4c-90bb-0be7f344f923" containerName="main" containerID="cri-o://7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57" Jul 02 13:32:13.512254 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:13.512053 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" podUID="25cd1f11-951d-4d4c-90bb-0be7f344f923" containerName="main" containerID="cri-o://7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57" gracePeriod=60 Jul 02 13:32:13.762141 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:13.762083 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:32:13.830186 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:13.830143 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs6tm\" (UniqueName: \"kubernetes.io/projected/25cd1f11-951d-4d4c-90bb-0be7f344f923-kube-api-access-rs6tm\") pod \"25cd1f11-951d-4d4c-90bb-0be7f344f923\" (UID: \"25cd1f11-951d-4d4c-90bb-0be7f344f923\") " Jul 02 13:32:13.830364 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:13.830224 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25cd1f11-951d-4d4c-90bb-0be7f344f923-tls-certs\") pod \"25cd1f11-951d-4d4c-90bb-0be7f344f923\" (UID: \"25cd1f11-951d-4d4c-90bb-0be7f344f923\") " Jul 02 13:32:13.830364 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:13.830322 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/25cd1f11-951d-4d4c-90bb-0be7f344f923-tokenizer-uds\") pod \"25cd1f11-951d-4d4c-90bb-0be7f344f923\" (UID: \"25cd1f11-951d-4d4c-90bb-0be7f344f923\") " Jul 02 13:32:13.830605 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:13.830590 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cd1f11-951d-4d4c-90bb-0be7f344f923-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "25cd1f11-951d-4d4c-90bb-0be7f344f923" (UID: "25cd1f11-951d-4d4c-90bb-0be7f344f923"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:32:13.832187 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:13.832166 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cd1f11-951d-4d4c-90bb-0be7f344f923-kube-api-access-rs6tm" (OuterVolumeSpecName: "kube-api-access-rs6tm") pod "25cd1f11-951d-4d4c-90bb-0be7f344f923" (UID: "25cd1f11-951d-4d4c-90bb-0be7f344f923"). InnerVolumeSpecName "kube-api-access-rs6tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 13:32:13.832249 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:13.832208 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25cd1f11-951d-4d4c-90bb-0be7f344f923-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25cd1f11-951d-4d4c-90bb-0be7f344f923" (UID: "25cd1f11-951d-4d4c-90bb-0be7f344f923"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 13:32:13.931760 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:13.931711 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25cd1f11-951d-4d4c-90bb-0be7f344f923-tls-certs\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:32:13.931760 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:13.931744 2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/25cd1f11-951d-4d4c-90bb-0be7f344f923-tokenizer-uds\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:32:13.931760 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:13.931754 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs6tm\" (UniqueName: \"kubernetes.io/projected/25cd1f11-951d-4d4c-90bb-0be7f344f923-kube-api-access-rs6tm\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:32:14.011818 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:14.011781 2578 generic.go:356] "Generic (PLEG): container finished" podID="25cd1f11-951d-4d4c-90bb-0be7f344f923" containerID="7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57" exitCode=0 Jul 02 13:32:14.012028 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:14.011845 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" Jul 02 13:32:14.012028 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:14.011867 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" event={"ID":"25cd1f11-951d-4d4c-90bb-0be7f344f923","Type":"ContainerDied","Data":"7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57"} Jul 02 13:32:14.012028 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:14.011974 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw" event={"ID":"25cd1f11-951d-4d4c-90bb-0be7f344f923","Type":"ContainerDied","Data":"60eb8368c9f3b2d7d11fb196b381b3643b4e7b554f4c8fe3d9c0894ca9437708"} Jul 02 13:32:14.012028 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:14.012004 2578 scope.go:117] "RemoveContainer" containerID="7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57" Jul 02 13:32:14.021573 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:14.021533 2578 scope.go:117] "RemoveContainer" containerID="7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57" Jul 02 13:32:14.021900 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:32:14.021874 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57\": container with ID starting with 7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57 not found: ID does not exist" containerID="7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57" Jul 02 13:32:14.022008 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:14.021905 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57"} err="failed to get container status \"7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57\": rpc error: code = NotFound desc = could not find container \"7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57\": container with ID starting with 7df3890f86165ad3589fc01554ba1fa9d4814300b93fd02c0820971055c33f57 not found: ID does not exist" Jul 02 13:32:14.034064 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:14.034031 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw"] Jul 02 13:32:14.039996 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:14.039975 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw"] Jul 02 13:32:15.627385 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:15.627354 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25cd1f11-951d-4d4c-90bb-0be7f344f923" path="/var/lib/kubelet/pods/25cd1f11-951d-4d4c-90bb-0be7f344f923/volumes" Jul 02 13:32:43.521677 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.521642 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8"] Jul 02 13:32:43.522101 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.522029 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25cd1f11-951d-4d4c-90bb-0be7f344f923" containerName="main" Jul 02 13:32:43.522101 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.522044 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="25cd1f11-951d-4d4c-90bb-0be7f344f923" containerName="main" Jul 02 13:32:43.522101 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.522055 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="106fb5e6-6779-444a-8d1c-0503a11a73ce" containerName="main" Jul 02 13:32:43.522101 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.522060 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="106fb5e6-6779-444a-8d1c-0503a11a73ce" containerName="main" Jul 02 13:32:43.522229 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.522124 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="106fb5e6-6779-444a-8d1c-0503a11a73ce" containerName="main" Jul 02 13:32:43.522229 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.522135 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="25cd1f11-951d-4d4c-90bb-0be7f344f923" containerName="main" Jul 02 13:32:43.524870 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.524853 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:32:43.527856 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.527833 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 02 13:32:43.529188 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.529168 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\"" Jul 02 13:32:43.529260 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.529181 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-6wxct\"" Jul 02 13:32:43.529317 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.529181 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\"" Jul 02 13:32:43.529317 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.529173 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 02 13:32:43.536429 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.536406 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8"] Jul 02 13:32:43.679016 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.678983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55m6\" (UniqueName: \"kubernetes.io/projected/2b90056b-307b-43f6-b632-de2254e0e9a6-kube-api-access-w55m6\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8\" (UID: \"2b90056b-307b-43f6-b632-de2254e0e9a6\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:32:43.679198 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.679035 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b90056b-307b-43f6-b632-de2254e0e9a6-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8\" (UID: \"2b90056b-307b-43f6-b632-de2254e0e9a6\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:32:43.679198 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.679111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b90056b-307b-43f6-b632-de2254e0e9a6-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8\" (UID: \"2b90056b-307b-43f6-b632-de2254e0e9a6\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:32:43.779639 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.779537 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55m6\" (UniqueName: \"kubernetes.io/projected/2b90056b-307b-43f6-b632-de2254e0e9a6-kube-api-access-w55m6\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8\" (UID: \"2b90056b-307b-43f6-b632-de2254e0e9a6\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:32:43.779639 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.779600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b90056b-307b-43f6-b632-de2254e0e9a6-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8\" (UID: \"2b90056b-307b-43f6-b632-de2254e0e9a6\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:32:43.779639 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.779640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b90056b-307b-43f6-b632-de2254e0e9a6-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8\" (UID: \"2b90056b-307b-43f6-b632-de2254e0e9a6\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:32:43.780086 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.780059 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b90056b-307b-43f6-b632-de2254e0e9a6-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8\" (UID: \"2b90056b-307b-43f6-b632-de2254e0e9a6\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:32:43.782142 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.782113 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b90056b-307b-43f6-b632-de2254e0e9a6-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8\" (UID: \"2b90056b-307b-43f6-b632-de2254e0e9a6\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:32:43.788580 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.788557 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55m6\" (UniqueName: \"kubernetes.io/projected/2b90056b-307b-43f6-b632-de2254e0e9a6-kube-api-access-w55m6\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8\" (UID: \"2b90056b-307b-43f6-b632-de2254e0e9a6\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:32:43.836074 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.836040 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:32:43.980712 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:43.980685 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8"] Jul 02 13:32:43.983433 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:32:43.983408 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b90056b_307b_43f6_b632_de2254e0e9a6.slice/crio-cf2ab584559d87dce903bbe1475cee2b26ca9025e7eb816e882195a47bd7992b WatchSource:0}: Error finding container cf2ab584559d87dce903bbe1475cee2b26ca9025e7eb816e882195a47bd7992b: Status 404 returned error can't find the container with id cf2ab584559d87dce903bbe1475cee2b26ca9025e7eb816e882195a47bd7992b Jul 02 13:32:44.135479 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:44.135432 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" event={"ID":"2b90056b-307b-43f6-b632-de2254e0e9a6","Type":"ContainerStarted","Data":"c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95"} Jul 02 13:32:44.135655 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:44.135501 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" event={"ID":"2b90056b-307b-43f6-b632-de2254e0e9a6","Type":"ContainerStarted","Data":"cf2ab584559d87dce903bbe1475cee2b26ca9025e7eb816e882195a47bd7992b"} Jul 02 13:32:44.135655 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:44.135551 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:32:44.154658 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:32:44.154602 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" podStartSLOduration=1.15458317 podStartE2EDuration="1.15458317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:32:44.154201307 +0000 UTC m=+1128.998914987" watchObservedRunningTime="2026-07-02 13:32:44.15458317 +0000 UTC m=+1128.999296847" Jul 02 13:33:15.141568 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:33:15.141540 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:33:55.627665 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:33:55.627638 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:33:55.628590 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:33:55.628567 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:34:18.371804 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.371771 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq"] Jul 02 13:34:18.375117 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.375097 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.378587 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.378567 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\"" Jul 02 13:34:18.387375 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.387352 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq"] Jul 02 13:34:18.463065 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.463038 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.463182 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.463072 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43f539a-eb38-478e-a35e-b60e04a23fac-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.463182 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.463091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.463182 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.463118 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.463293 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.463186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5sbj\" (UniqueName: \"kubernetes.io/projected/a43f539a-eb38-478e-a35e-b60e04a23fac-kube-api-access-s5sbj\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.463293 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.463219 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.564467 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.564442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.564596 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.564491 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5sbj\" (UniqueName: \"kubernetes.io/projected/a43f539a-eb38-478e-a35e-b60e04a23fac-kube-api-access-s5sbj\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.564596 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.564521 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.564596 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.564559 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.564737 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.564687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43f539a-eb38-478e-a35e-b60e04a23fac-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.564789 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.564734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.564977 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.564957 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.565068 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.564984 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.565125 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.565089 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.566796 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.566777 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.567195 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.567176 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43f539a-eb38-478e-a35e-b60e04a23fac-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.573140 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.573115 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5sbj\" (UniqueName: \"kubernetes.io/projected/a43f539a-eb38-478e-a35e-b60e04a23fac-kube-api-access-s5sbj\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.684748 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.684673 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:18.815299 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:18.815275 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq"] Jul 02 13:34:18.817393 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:34:18.817356 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43f539a_eb38_478e_a35e_b60e04a23fac.slice/crio-dda27c90e8ab2d6c5583eb613e758b22a742460ea7396b9435ef8164f98e9a06 WatchSource:0}: Error finding container dda27c90e8ab2d6c5583eb613e758b22a742460ea7396b9435ef8164f98e9a06: Status 404 returned error can't find the container with id dda27c90e8ab2d6c5583eb613e758b22a742460ea7396b9435ef8164f98e9a06 Jul 02 13:34:19.545459 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:19.545409 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" event={"ID":"a43f539a-eb38-478e-a35e-b60e04a23fac","Type":"ContainerStarted","Data":"13b5719da9782c26117206b0e454b98c3847fcf15ca1abc7b9ad5bb111861b38"} Jul 02 13:34:19.545870 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:19.545482 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" event={"ID":"a43f539a-eb38-478e-a35e-b60e04a23fac","Type":"ContainerStarted","Data":"dda27c90e8ab2d6c5583eb613e758b22a742460ea7396b9435ef8164f98e9a06"} Jul 02 13:34:19.567147 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:19.567087 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" podStartSLOduration=1.567070892 podStartE2EDuration="1.567070892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:34:19.566480174 +0000 UTC m=+1224.411193852" watchObservedRunningTime="2026-07-02 13:34:19.567070892 +0000 UTC m=+1224.411784570" Jul 02 13:34:19.685199 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:19.685166 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:28.684812 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:28.684774 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:28.697298 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:28.697269 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:29.600025 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:29.599997 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:34:52.253565 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:34:52.253482 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq"] Jul 02 13:35:07.274175 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.274126 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" podUID="a43f539a-eb38-478e-a35e-b60e04a23fac" containerName="main" containerID="cri-o://13b5719da9782c26117206b0e454b98c3847fcf15ca1abc7b9ad5bb111861b38" gracePeriod=45 Jul 02 13:35:07.520361 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.520329 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:35:07.576731 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.576702 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5sbj\" (UniqueName: \"kubernetes.io/projected/a43f539a-eb38-478e-a35e-b60e04a23fac-kube-api-access-s5sbj\") pod \"a43f539a-eb38-478e-a35e-b60e04a23fac\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " Jul 02 13:35:07.576912 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.576741 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43f539a-eb38-478e-a35e-b60e04a23fac-tls-certs\") pod \"a43f539a-eb38-478e-a35e-b60e04a23fac\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " Jul 02 13:35:07.576912 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.576764 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-tmp-dir\") pod \"a43f539a-eb38-478e-a35e-b60e04a23fac\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " Jul 02 13:35:07.576912 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.576838 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-home\") pod \"a43f539a-eb38-478e-a35e-b60e04a23fac\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " Jul 02 13:35:07.576912 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.576869 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-model-cache\") pod \"a43f539a-eb38-478e-a35e-b60e04a23fac\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " Jul 02 13:35:07.577235 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.576950 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-dshm\") pod \"a43f539a-eb38-478e-a35e-b60e04a23fac\" (UID: \"a43f539a-eb38-478e-a35e-b60e04a23fac\") " Jul 02 13:35:07.577235 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.577058 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a43f539a-eb38-478e-a35e-b60e04a23fac" (UID: "a43f539a-eb38-478e-a35e-b60e04a23fac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:35:07.577235 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.577182 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-home" (OuterVolumeSpecName: "home") pod "a43f539a-eb38-478e-a35e-b60e04a23fac" (UID: "a43f539a-eb38-478e-a35e-b60e04a23fac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:35:07.577235 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.577207 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-tmp-dir\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:35:07.577235 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.577225 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-model-cache" (OuterVolumeSpecName: "model-cache") pod "a43f539a-eb38-478e-a35e-b60e04a23fac" (UID: "a43f539a-eb38-478e-a35e-b60e04a23fac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:35:07.579079 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.579048 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a43f539a-eb38-478e-a35e-b60e04a23fac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a43f539a-eb38-478e-a35e-b60e04a23fac" (UID: "a43f539a-eb38-478e-a35e-b60e04a23fac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 13:35:07.579079 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.579052 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-dshm" (OuterVolumeSpecName: "dshm") pod "a43f539a-eb38-478e-a35e-b60e04a23fac" (UID: "a43f539a-eb38-478e-a35e-b60e04a23fac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:35:07.579228 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.579112 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43f539a-eb38-478e-a35e-b60e04a23fac-kube-api-access-s5sbj" (OuterVolumeSpecName: "kube-api-access-s5sbj") pod "a43f539a-eb38-478e-a35e-b60e04a23fac" (UID: "a43f539a-eb38-478e-a35e-b60e04a23fac"). InnerVolumeSpecName "kube-api-access-s5sbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 13:35:07.678412 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.678380 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-model-cache\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:35:07.678412 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.678405 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-dshm\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:35:07.678611 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.678436 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5sbj\" (UniqueName: \"kubernetes.io/projected/a43f539a-eb38-478e-a35e-b60e04a23fac-kube-api-access-s5sbj\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:35:07.678611 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.678445 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43f539a-eb38-478e-a35e-b60e04a23fac-tls-certs\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:35:07.678611 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.678455 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43f539a-eb38-478e-a35e-b60e04a23fac-home\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:35:07.754146 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.754112 2578 generic.go:356] "Generic (PLEG): container finished" podID="a43f539a-eb38-478e-a35e-b60e04a23fac" containerID="13b5719da9782c26117206b0e454b98c3847fcf15ca1abc7b9ad5bb111861b38" exitCode=0 Jul 02 13:35:07.754310 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.754177 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" event={"ID":"a43f539a-eb38-478e-a35e-b60e04a23fac","Type":"ContainerDied","Data":"13b5719da9782c26117206b0e454b98c3847fcf15ca1abc7b9ad5bb111861b38"} Jul 02 13:35:07.754310 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.754243 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" event={"ID":"a43f539a-eb38-478e-a35e-b60e04a23fac","Type":"ContainerDied","Data":"dda27c90e8ab2d6c5583eb613e758b22a742460ea7396b9435ef8164f98e9a06"} Jul 02 13:35:07.754310 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.754179 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq" Jul 02 13:35:07.754310 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.754266 2578 scope.go:117] "RemoveContainer" containerID="13b5719da9782c26117206b0e454b98c3847fcf15ca1abc7b9ad5bb111861b38" Jul 02 13:35:07.762425 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.762405 2578 scope.go:117] "RemoveContainer" containerID="13b5719da9782c26117206b0e454b98c3847fcf15ca1abc7b9ad5bb111861b38" Jul 02 13:35:07.762702 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:35:07.762679 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b5719da9782c26117206b0e454b98c3847fcf15ca1abc7b9ad5bb111861b38\": container with ID starting with 13b5719da9782c26117206b0e454b98c3847fcf15ca1abc7b9ad5bb111861b38 not found: ID does not exist" containerID="13b5719da9782c26117206b0e454b98c3847fcf15ca1abc7b9ad5bb111861b38" Jul 02 13:35:07.762750 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.762710 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b5719da9782c26117206b0e454b98c3847fcf15ca1abc7b9ad5bb111861b38"} err="failed to get container status \"13b5719da9782c26117206b0e454b98c3847fcf15ca1abc7b9ad5bb111861b38\": rpc error: code = NotFound desc = could not find container \"13b5719da9782c26117206b0e454b98c3847fcf15ca1abc7b9ad5bb111861b38\": container with ID starting with 13b5719da9782c26117206b0e454b98c3847fcf15ca1abc7b9ad5bb111861b38 not found: ID does not exist" Jul 02 13:35:07.772999 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.772975 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq"] Jul 02 13:35:07.777666 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:07.777643 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq"] Jul 02 13:35:09.624655 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:09.624623 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a43f539a-eb38-478e-a35e-b60e04a23fac" path="/var/lib/kubelet/pods/a43f539a-eb38-478e-a35e-b60e04a23fac/volumes" Jul 02 13:35:12.469627 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.469593 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8"] Jul 02 13:35:12.478496 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:35:12.478453 2578 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 02 13:35:12.478496 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 13:35:12.478496 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 13:35:12.478496 ip-10-0-131-243 kubenswrapper[2578]: > containerID="c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95" cmd=["/bin/sleep","15"] Jul 02 13:35:12.478741 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.478518 2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" message="" err=< Jul 02 13:35:12.478741 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 13:35:12.478741 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 13:35:12.478741 ip-10-0-131-243 kubenswrapper[2578]: > Jul 02 13:35:12.478741 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:35:12.478534 2578 kuberuntime_container.go:754] "PreStop hook failed" err=< Jul 02 13:35:12.478741 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 13:35:12.478741 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 13:35:12.478741 ip-10-0-131-243 kubenswrapper[2578]: > pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" podUID="2b90056b-307b-43f6-b632-de2254e0e9a6" containerName="main" containerID="cri-o://c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95" Jul 02 13:35:12.478741 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.478589 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" podUID="2b90056b-307b-43f6-b632-de2254e0e9a6" containerName="main" containerID="cri-o://c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95" gracePeriod=60 Jul 02 13:35:12.730146 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.730087 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:35:12.777744 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.777710 2578 generic.go:356] "Generic (PLEG): container finished" podID="2b90056b-307b-43f6-b632-de2254e0e9a6" containerID="c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95" exitCode=0 Jul 02 13:35:12.777900 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.777752 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" event={"ID":"2b90056b-307b-43f6-b632-de2254e0e9a6","Type":"ContainerDied","Data":"c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95"} Jul 02 13:35:12.777900 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.777756 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" Jul 02 13:35:12.777900 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.777788 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8" event={"ID":"2b90056b-307b-43f6-b632-de2254e0e9a6","Type":"ContainerDied","Data":"cf2ab584559d87dce903bbe1475cee2b26ca9025e7eb816e882195a47bd7992b"} Jul 02 13:35:12.777900 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.777817 2578 scope.go:117] "RemoveContainer" containerID="c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95" Jul 02 13:35:12.785720 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.785703 2578 scope.go:117] "RemoveContainer" containerID="c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95" Jul 02 13:35:12.785976 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:35:12.785955 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95\": container with ID starting with c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95 not found: ID does not exist" containerID="c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95" Jul 02 13:35:12.786034 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.785983 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95"} err="failed to get container status \"c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95\": rpc error: code = NotFound desc = could not find container \"c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95\": container with ID starting with c35e4540cc35697c306857f8b1d3d40f3e82f9262d448db0357de6be7608cf95 not found: ID does not exist" Jul 02 13:35:12.821351 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.821327 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b90056b-307b-43f6-b632-de2254e0e9a6-tokenizer-uds\") pod \"2b90056b-307b-43f6-b632-de2254e0e9a6\" (UID: \"2b90056b-307b-43f6-b632-de2254e0e9a6\") " Jul 02 13:35:12.821433 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.821361 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w55m6\" (UniqueName: \"kubernetes.io/projected/2b90056b-307b-43f6-b632-de2254e0e9a6-kube-api-access-w55m6\") pod \"2b90056b-307b-43f6-b632-de2254e0e9a6\" (UID: \"2b90056b-307b-43f6-b632-de2254e0e9a6\") " Jul 02 13:35:12.821484 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.821447 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b90056b-307b-43f6-b632-de2254e0e9a6-tls-certs\") pod \"2b90056b-307b-43f6-b632-de2254e0e9a6\" (UID: \"2b90056b-307b-43f6-b632-de2254e0e9a6\") " Jul 02 13:35:12.821610 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.821590 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b90056b-307b-43f6-b632-de2254e0e9a6-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2b90056b-307b-43f6-b632-de2254e0e9a6" (UID: "2b90056b-307b-43f6-b632-de2254e0e9a6"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:35:12.821711 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.821699 2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b90056b-307b-43f6-b632-de2254e0e9a6-tokenizer-uds\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:35:12.823437 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.823408 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b90056b-307b-43f6-b632-de2254e0e9a6-kube-api-access-w55m6" (OuterVolumeSpecName: "kube-api-access-w55m6") pod "2b90056b-307b-43f6-b632-de2254e0e9a6" (UID: "2b90056b-307b-43f6-b632-de2254e0e9a6"). InnerVolumeSpecName "kube-api-access-w55m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 13:35:12.823517 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.823440 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b90056b-307b-43f6-b632-de2254e0e9a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b90056b-307b-43f6-b632-de2254e0e9a6" (UID: "2b90056b-307b-43f6-b632-de2254e0e9a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 13:35:12.922530 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.922482 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b90056b-307b-43f6-b632-de2254e0e9a6-tls-certs\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:35:12.922530 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:12.922526 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w55m6\" (UniqueName: \"kubernetes.io/projected/2b90056b-307b-43f6-b632-de2254e0e9a6-kube-api-access-w55m6\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:35:13.099554 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:13.099522 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8"] Jul 02 13:35:13.103139 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:13.103116 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8"] Jul 02 13:35:13.624890 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:35:13.624860 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b90056b-307b-43f6-b632-de2254e0e9a6" path="/var/lib/kubelet/pods/2b90056b-307b-43f6-b632-de2254e0e9a6/volumes" Jul 02 13:36:30.987633 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:30.987597 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5"] Jul 02 13:36:30.988280 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:30.987936 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b90056b-307b-43f6-b632-de2254e0e9a6" containerName="main" Jul 02 13:36:30.988280 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:30.987951 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b90056b-307b-43f6-b632-de2254e0e9a6" containerName="main" Jul 02 13:36:30.988280 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:30.987962 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43f539a-eb38-478e-a35e-b60e04a23fac" containerName="main" Jul 02 13:36:30.988280 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:30.987967 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43f539a-eb38-478e-a35e-b60e04a23fac" containerName="main" Jul 02 13:36:30.988280 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:30.988040 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43f539a-eb38-478e-a35e-b60e04a23fac" containerName="main" Jul 02 13:36:30.988280 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:30.988053 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b90056b-307b-43f6-b632-de2254e0e9a6" containerName="main" Jul 02 13:36:30.991093 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:30.991070 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:30.995465 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:30.995097 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\"" Jul 02 13:36:30.995465 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:30.995196 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\"" Jul 02 13:36:30.995465 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:30.995284 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 02 13:36:30.995465 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:30.995357 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 02 13:36:31.002582 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.002553 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5"] Jul 02 13:36:31.090294 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.090265 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.090457 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.090306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.090457 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.090327 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.090457 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.090378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.090457 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.090395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w796\" (UniqueName: \"kubernetes.io/projected/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-kube-api-access-4w796\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.090457 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.090439 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.190786 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.190747 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.191004 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.190801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.191004 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.190829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.191004 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.190876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.191004 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.190900 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w796\" (UniqueName: \"kubernetes.io/projected/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-kube-api-access-4w796\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.191004 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.190961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.191296 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.191252 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.191296 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.191274 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.191400 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.191317 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.193204 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.193178 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.193555 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.193534 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.200115 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.200088 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w796\" (UniqueName: \"kubernetes.io/projected/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-kube-api-access-4w796\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.304170 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.304082 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:31.436209 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:31.436186 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5"] Jul 02 13:36:31.438462 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:36:31.438436 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1a7994_9cd2_49ab_89fe_0293b8fd35d1.slice/crio-7cbe920cbc229dec0a3fb5ed62290e539243d96fadcbe8bb89c3f4295ef3d2e2 WatchSource:0}: Error finding container 7cbe920cbc229dec0a3fb5ed62290e539243d96fadcbe8bb89c3f4295ef3d2e2: Status 404 returned error can't find the container with id 7cbe920cbc229dec0a3fb5ed62290e539243d96fadcbe8bb89c3f4295ef3d2e2 Jul 02 13:36:32.120339 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:32.120304 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" event={"ID":"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1","Type":"ContainerStarted","Data":"ebaf5da58e455aa6e74f0b457f7ca1342024d12dd1b0a47b522241dc6c305690"} Jul 02 13:36:32.120742 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:32.120361 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" event={"ID":"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1","Type":"ContainerStarted","Data":"7cbe920cbc229dec0a3fb5ed62290e539243d96fadcbe8bb89c3f4295ef3d2e2"} Jul 02 13:36:32.138291 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:32.138252 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" podStartSLOduration=2.138239623 podStartE2EDuration="2.138239623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:36:32.137133317 +0000 UTC m=+1356.981846995" watchObservedRunningTime="2026-07-02 13:36:32.138239623 +0000 UTC m=+1356.982953300" Jul 02 13:36:32.304959 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:32.304931 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:41.304715 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:41.304674 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:41.317417 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:41.317390 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:42.173064 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:42.173028 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:43.453315 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:43.453278 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5"] Jul 02 13:36:59.189790 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.189740 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" podUID="6e1a7994-9cd2-49ab-89fe-0293b8fd35d1" containerName="main" containerID="cri-o://ebaf5da58e455aa6e74f0b457f7ca1342024d12dd1b0a47b522241dc6c305690" gracePeriod=45 Jul 02 13:36:59.435945 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.435901 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:36:59.536941 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.536829 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-tmp-dir\") pod \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " Jul 02 13:36:59.536941 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.536868 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-home\") pod \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " Jul 02 13:36:59.536941 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.536890 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w796\" (UniqueName: \"kubernetes.io/projected/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-kube-api-access-4w796\") pod \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " Jul 02 13:36:59.537236 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.536955 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-model-cache\") pod \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " Jul 02 13:36:59.537236 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.536984 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-tls-certs\") pod \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " Jul 02 13:36:59.537236 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.537042 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-dshm\") pod \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\" (UID: \"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1\") " Jul 02 13:36:59.537236 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.537212 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-home" (OuterVolumeSpecName: "home") pod "6e1a7994-9cd2-49ab-89fe-0293b8fd35d1" (UID: "6e1a7994-9cd2-49ab-89fe-0293b8fd35d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:36:59.537419 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.537229 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e1a7994-9cd2-49ab-89fe-0293b8fd35d1" (UID: "6e1a7994-9cd2-49ab-89fe-0293b8fd35d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:36:59.537419 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.537237 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e1a7994-9cd2-49ab-89fe-0293b8fd35d1" (UID: "6e1a7994-9cd2-49ab-89fe-0293b8fd35d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:36:59.539133 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.539107 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-dshm" (OuterVolumeSpecName: "dshm") pod "6e1a7994-9cd2-49ab-89fe-0293b8fd35d1" (UID: "6e1a7994-9cd2-49ab-89fe-0293b8fd35d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:36:59.539566 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.539545 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e1a7994-9cd2-49ab-89fe-0293b8fd35d1" (UID: "6e1a7994-9cd2-49ab-89fe-0293b8fd35d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 13:36:59.539673 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.539561 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-kube-api-access-4w796" (OuterVolumeSpecName: "kube-api-access-4w796") pod "6e1a7994-9cd2-49ab-89fe-0293b8fd35d1" (UID: "6e1a7994-9cd2-49ab-89fe-0293b8fd35d1"). InnerVolumeSpecName "kube-api-access-4w796". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 13:36:59.638490 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.638465 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-model-cache\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:36:59.638490 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.638490 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-tls-certs\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:36:59.638636 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.638498 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-dshm\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:36:59.638636 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.638507 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-tmp-dir\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:36:59.638636 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.638514 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-home\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:36:59.638636 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:36:59.638522 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w796\" (UniqueName: \"kubernetes.io/projected/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1-kube-api-access-4w796\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:37:00.243334 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:37:00.243297 2578 generic.go:356] "Generic (PLEG): container finished" podID="6e1a7994-9cd2-49ab-89fe-0293b8fd35d1" containerID="ebaf5da58e455aa6e74f0b457f7ca1342024d12dd1b0a47b522241dc6c305690" exitCode=0 Jul 02 13:37:00.243756 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:37:00.243347 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" Jul 02 13:37:00.243756 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:37:00.243384 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" event={"ID":"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1","Type":"ContainerDied","Data":"ebaf5da58e455aa6e74f0b457f7ca1342024d12dd1b0a47b522241dc6c305690"} Jul 02 13:37:00.243756 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:37:00.243445 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5" event={"ID":"6e1a7994-9cd2-49ab-89fe-0293b8fd35d1","Type":"ContainerDied","Data":"7cbe920cbc229dec0a3fb5ed62290e539243d96fadcbe8bb89c3f4295ef3d2e2"} Jul 02 13:37:00.243756 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:37:00.243471 2578 scope.go:117] "RemoveContainer" containerID="ebaf5da58e455aa6e74f0b457f7ca1342024d12dd1b0a47b522241dc6c305690" Jul 02 13:37:00.251575 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:37:00.251555 2578 scope.go:117] "RemoveContainer" containerID="ebaf5da58e455aa6e74f0b457f7ca1342024d12dd1b0a47b522241dc6c305690" Jul 02 13:37:00.251824 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:37:00.251808 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebaf5da58e455aa6e74f0b457f7ca1342024d12dd1b0a47b522241dc6c305690\": container with ID starting with ebaf5da58e455aa6e74f0b457f7ca1342024d12dd1b0a47b522241dc6c305690 not found: ID does not exist" containerID="ebaf5da58e455aa6e74f0b457f7ca1342024d12dd1b0a47b522241dc6c305690" Jul 02 13:37:00.251877 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:37:00.251831 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebaf5da58e455aa6e74f0b457f7ca1342024d12dd1b0a47b522241dc6c305690"} err="failed to get container status \"ebaf5da58e455aa6e74f0b457f7ca1342024d12dd1b0a47b522241dc6c305690\": rpc error: code = NotFound desc = could not find container \"ebaf5da58e455aa6e74f0b457f7ca1342024d12dd1b0a47b522241dc6c305690\": container with ID starting with ebaf5da58e455aa6e74f0b457f7ca1342024d12dd1b0a47b522241dc6c305690 not found: ID does not exist" Jul 02 13:37:00.262686 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:37:00.262664 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5"] Jul 02 13:37:00.269434 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:37:00.269411 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5"] Jul 02 13:37:01.624596 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:37:01.624562 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e1a7994-9cd2-49ab-89fe-0293b8fd35d1" path="/var/lib/kubelet/pods/6e1a7994-9cd2-49ab-89fe-0293b8fd35d1/volumes" Jul 02 13:38:55.651723 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:38:55.651691 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:38:55.652951 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:38:55.652910 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:40:17.436595 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.436562 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt"] Jul 02 13:40:17.437083 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.436866 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1a7994-9cd2-49ab-89fe-0293b8fd35d1" containerName="main" Jul 02 13:40:17.437083 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.436878 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1a7994-9cd2-49ab-89fe-0293b8fd35d1" containerName="main" Jul 02 13:40:17.437083 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.436956 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e1a7994-9cd2-49ab-89fe-0293b8fd35d1" containerName="main" Jul 02 13:40:17.439763 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.439747 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:40:17.443539 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.443509 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-jrvpj\"" Jul 02 13:40:17.443691 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.443557 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 02 13:40:17.443691 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.443583 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\"" Jul 02 13:40:17.443691 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.443563 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 02 13:40:17.443691 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.443559 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\"" Jul 02 13:40:17.449850 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.449830 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt"] Jul 02 13:40:17.600098 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.600063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzs4v\" (UniqueName: \"kubernetes.io/projected/ccddc9cc-f129-4519-8848-ee660cb7897d-kube-api-access-jzs4v\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt\" (UID: \"ccddc9cc-f129-4519-8848-ee660cb7897d\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:40:17.600267 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.600110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ccddc9cc-f129-4519-8848-ee660cb7897d-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt\" (UID: \"ccddc9cc-f129-4519-8848-ee660cb7897d\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:40:17.600267 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.600132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccddc9cc-f129-4519-8848-ee660cb7897d-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt\" (UID: \"ccddc9cc-f129-4519-8848-ee660cb7897d\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:40:17.700739 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.700661 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzs4v\" (UniqueName: \"kubernetes.io/projected/ccddc9cc-f129-4519-8848-ee660cb7897d-kube-api-access-jzs4v\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt\" (UID: \"ccddc9cc-f129-4519-8848-ee660cb7897d\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:40:17.700739 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.700712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ccddc9cc-f129-4519-8848-ee660cb7897d-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt\" (UID: \"ccddc9cc-f129-4519-8848-ee660cb7897d\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:40:17.700940 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.700748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccddc9cc-f129-4519-8848-ee660cb7897d-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt\" (UID: \"ccddc9cc-f129-4519-8848-ee660cb7897d\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:40:17.701170 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.701148 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ccddc9cc-f129-4519-8848-ee660cb7897d-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt\" (UID: \"ccddc9cc-f129-4519-8848-ee660cb7897d\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:40:17.703305 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.703284 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccddc9cc-f129-4519-8848-ee660cb7897d-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt\" (UID: \"ccddc9cc-f129-4519-8848-ee660cb7897d\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:40:17.708393 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.708367 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzs4v\" (UniqueName: \"kubernetes.io/projected/ccddc9cc-f129-4519-8848-ee660cb7897d-kube-api-access-jzs4v\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt\" (UID: \"ccddc9cc-f129-4519-8848-ee660cb7897d\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:40:17.749311 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.749283 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:40:17.879478 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:17.879352 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt"] Jul 02 13:40:17.882358 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:40:17.882331 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccddc9cc_f129_4519_8848_ee660cb7897d.slice/crio-499d9400d94cbc568b46f05c47d2c60c64bf3f8c9711426d0c8279436ec475a6 WatchSource:0}: Error finding container 499d9400d94cbc568b46f05c47d2c60c64bf3f8c9711426d0c8279436ec475a6: Status 404 returned error can't find the container with id 499d9400d94cbc568b46f05c47d2c60c64bf3f8c9711426d0c8279436ec475a6 Jul 02 13:40:18.092365 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:18.092330 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" event={"ID":"ccddc9cc-f129-4519-8848-ee660cb7897d","Type":"ContainerStarted","Data":"2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c"} Jul 02 13:40:18.092551 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:18.092387 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" event={"ID":"ccddc9cc-f129-4519-8848-ee660cb7897d","Type":"ContainerStarted","Data":"499d9400d94cbc568b46f05c47d2c60c64bf3f8c9711426d0c8279436ec475a6"} Jul 02 13:40:18.092551 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:18.092414 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:40:49.098393 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:49.098320 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:40:49.118339 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:40:49.118292 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" podStartSLOduration=32.118276168 podStartE2EDuration="32.118276168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:40:18.12678571 +0000 UTC m=+1582.971499406" watchObservedRunningTime="2026-07-02 13:40:49.118276168 +0000 UTC m=+1613.962989847" Jul 02 13:42:55.988174 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:55.988141 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt"] Jul 02 13:42:55.996911 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:42:55.996876 2578 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 02 13:42:55.996911 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 13:42:55.996911 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 13:42:55.996911 ip-10-0-131-243 kubenswrapper[2578]: > containerID="2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c" cmd=["/bin/sleep","15"] Jul 02 13:42:55.997208 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:55.996954 2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" message="" err=< Jul 02 13:42:55.997208 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 13:42:55.997208 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 13:42:55.997208 ip-10-0-131-243 kubenswrapper[2578]: > Jul 02 13:42:55.997208 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:42:55.996971 2578 kuberuntime_container.go:754] "PreStop hook failed" err=< Jul 02 13:42:55.997208 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 13:42:55.997208 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 13:42:55.997208 ip-10-0-131-243 kubenswrapper[2578]: > pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" podUID="ccddc9cc-f129-4519-8848-ee660cb7897d" containerName="main" containerID="cri-o://2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c" Jul 02 13:42:55.997208 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:55.997024 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" podUID="ccddc9cc-f129-4519-8848-ee660cb7897d" containerName="main" containerID="cri-o://2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c" gracePeriod=60 Jul 02 13:42:56.245906 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.245849 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:42:56.405497 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.405467 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccddc9cc-f129-4519-8848-ee660cb7897d-tls-certs\") pod \"ccddc9cc-f129-4519-8848-ee660cb7897d\" (UID: \"ccddc9cc-f129-4519-8848-ee660cb7897d\") " Jul 02 13:42:56.405672 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.405513 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzs4v\" (UniqueName: \"kubernetes.io/projected/ccddc9cc-f129-4519-8848-ee660cb7897d-kube-api-access-jzs4v\") pod \"ccddc9cc-f129-4519-8848-ee660cb7897d\" (UID: \"ccddc9cc-f129-4519-8848-ee660cb7897d\") " Jul 02 13:42:56.405672 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.405589 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ccddc9cc-f129-4519-8848-ee660cb7897d-tokenizer-uds\") pod \"ccddc9cc-f129-4519-8848-ee660cb7897d\" (UID: \"ccddc9cc-f129-4519-8848-ee660cb7897d\") " Jul 02 13:42:56.405896 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.405871 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccddc9cc-f129-4519-8848-ee660cb7897d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ccddc9cc-f129-4519-8848-ee660cb7897d" (UID: "ccddc9cc-f129-4519-8848-ee660cb7897d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 13:42:56.407518 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.407489 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccddc9cc-f129-4519-8848-ee660cb7897d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ccddc9cc-f129-4519-8848-ee660cb7897d" (UID: "ccddc9cc-f129-4519-8848-ee660cb7897d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 13:42:56.407610 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.407587 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccddc9cc-f129-4519-8848-ee660cb7897d-kube-api-access-jzs4v" (OuterVolumeSpecName: "kube-api-access-jzs4v") pod "ccddc9cc-f129-4519-8848-ee660cb7897d" (UID: "ccddc9cc-f129-4519-8848-ee660cb7897d"). InnerVolumeSpecName "kube-api-access-jzs4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 13:42:56.507069 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.507007 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccddc9cc-f129-4519-8848-ee660cb7897d-tls-certs\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:42:56.507069 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.507032 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzs4v\" (UniqueName: \"kubernetes.io/projected/ccddc9cc-f129-4519-8848-ee660cb7897d-kube-api-access-jzs4v\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:42:56.507069 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.507042 2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ccddc9cc-f129-4519-8848-ee660cb7897d-tokenizer-uds\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 13:42:56.757154 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.757074 2578 generic.go:356] "Generic (PLEG): container finished" podID="ccddc9cc-f129-4519-8848-ee660cb7897d" containerID="2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c" exitCode=0 Jul 02 13:42:56.757154 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.757114 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" Jul 02 13:42:56.757330 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.757113 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" event={"ID":"ccddc9cc-f129-4519-8848-ee660cb7897d","Type":"ContainerDied","Data":"2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c"} Jul 02 13:42:56.757330 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.757246 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt" event={"ID":"ccddc9cc-f129-4519-8848-ee660cb7897d","Type":"ContainerDied","Data":"499d9400d94cbc568b46f05c47d2c60c64bf3f8c9711426d0c8279436ec475a6"} Jul 02 13:42:56.757330 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.757274 2578 scope.go:117] "RemoveContainer" containerID="2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c" Jul 02 13:42:56.766353 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.766336 2578 scope.go:117] "RemoveContainer" containerID="2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c" Jul 02 13:42:56.766588 ip-10-0-131-243 kubenswrapper[2578]: E0702 13:42:56.766570 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c\": container with ID starting with 2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c not found: ID does not exist" containerID="2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c" Jul 02 13:42:56.766656 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.766594 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c"} err="failed to get container status \"2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c\": rpc error: code = NotFound desc = could not find container \"2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c\": container with ID starting with 2a1cbbfc418effa88dd7d079ddac10224f9bea294fcf4fea3a592437f8a9311c not found: ID does not exist" Jul 02 13:42:56.782203 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.782178 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt"] Jul 02 13:42:56.788189 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:56.788167 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt"] Jul 02 13:42:57.625158 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:42:57.625125 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccddc9cc-f129-4519-8848-ee660cb7897d" path="/var/lib/kubelet/pods/ccddc9cc-f129-4519-8848-ee660cb7897d/volumes" Jul 02 13:43:27.212456 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.212422 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd"] Jul 02 13:43:27.212927 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.212730 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccddc9cc-f129-4519-8848-ee660cb7897d" containerName="main" Jul 02 13:43:27.212927 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.212740 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccddc9cc-f129-4519-8848-ee660cb7897d" containerName="main" Jul 02 13:43:27.212927 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.212791 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccddc9cc-f129-4519-8848-ee660cb7897d" containerName="main" Jul 02 13:43:27.215460 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.215438 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" Jul 02 13:43:27.219177 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.219137 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 02 13:43:27.219177 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.219171 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\"" Jul 02 13:43:27.219368 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.219252 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 02 13:43:27.219368 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.219274 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-cp7w2\"" Jul 02 13:43:27.219468 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.219410 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\"" Jul 02 13:43:27.227015 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.226989 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd"] Jul 02 13:43:27.248022 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.247993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjbxl\" (UniqueName: \"kubernetes.io/projected/3dd126ed-b241-4f3f-bc01-3e3abe1b7a01-kube-api-access-kjbxl\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd\" (UID: \"3dd126ed-b241-4f3f-bc01-3e3abe1b7a01\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" Jul 02 13:43:27.248123 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.248030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd126ed-b241-4f3f-bc01-3e3abe1b7a01-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd\" (UID: \"3dd126ed-b241-4f3f-bc01-3e3abe1b7a01\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" Jul 02 13:43:27.248123 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.248078 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3dd126ed-b241-4f3f-bc01-3e3abe1b7a01-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd\" (UID: \"3dd126ed-b241-4f3f-bc01-3e3abe1b7a01\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" Jul 02 13:43:27.348433 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.348405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjbxl\" (UniqueName: \"kubernetes.io/projected/3dd126ed-b241-4f3f-bc01-3e3abe1b7a01-kube-api-access-kjbxl\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd\" (UID: \"3dd126ed-b241-4f3f-bc01-3e3abe1b7a01\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" Jul 02 13:43:27.348540 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.348442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd126ed-b241-4f3f-bc01-3e3abe1b7a01-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd\" (UID: \"3dd126ed-b241-4f3f-bc01-3e3abe1b7a01\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" Jul 02 13:43:27.348540 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.348476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3dd126ed-b241-4f3f-bc01-3e3abe1b7a01-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd\" (UID: \"3dd126ed-b241-4f3f-bc01-3e3abe1b7a01\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" Jul 02 13:43:27.348889 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.348870 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3dd126ed-b241-4f3f-bc01-3e3abe1b7a01-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd\" (UID: \"3dd126ed-b241-4f3f-bc01-3e3abe1b7a01\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" Jul 02 13:43:27.350755 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.350733 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd126ed-b241-4f3f-bc01-3e3abe1b7a01-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd\" (UID: \"3dd126ed-b241-4f3f-bc01-3e3abe1b7a01\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" Jul 02 13:43:27.356549 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.356530 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjbxl\" (UniqueName: \"kubernetes.io/projected/3dd126ed-b241-4f3f-bc01-3e3abe1b7a01-kube-api-access-kjbxl\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd\" (UID: \"3dd126ed-b241-4f3f-bc01-3e3abe1b7a01\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" Jul 02 13:43:27.526240 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.526134 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" Jul 02 13:43:27.657231 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.657196 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd"] Jul 02 13:43:27.660723 ip-10-0-131-243 kubenswrapper[2578]: W0702 13:43:27.660688 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd126ed_b241_4f3f_bc01_3e3abe1b7a01.slice/crio-8c0891749c2ed2ad7009f5c53873e8b89c26d0c9be721a0a46f02287dc0be874 WatchSource:0}: Error finding container 8c0891749c2ed2ad7009f5c53873e8b89c26d0c9be721a0a46f02287dc0be874: Status 404 returned error can't find the container with id 8c0891749c2ed2ad7009f5c53873e8b89c26d0c9be721a0a46f02287dc0be874 Jul 02 13:43:27.900322 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.900287 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" event={"ID":"3dd126ed-b241-4f3f-bc01-3e3abe1b7a01","Type":"ContainerStarted","Data":"f4d7b2e3d44cd70ecc5b2dfc4a6428466df7a2a382e6384ad0f871dee1df6203"} Jul 02 13:43:27.900479 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.900336 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" event={"ID":"3dd126ed-b241-4f3f-bc01-3e3abe1b7a01","Type":"ContainerStarted","Data":"8c0891749c2ed2ad7009f5c53873e8b89c26d0c9be721a0a46f02287dc0be874"} Jul 02 13:43:27.900479 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.900365 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" Jul 02 13:43:27.917897 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:27.917844 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" podStartSLOduration=0.917824054 podStartE2EDuration="917.824054ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:43:27.916888071 +0000 UTC m=+1772.761601768" watchObservedRunningTime="2026-07-02 13:43:27.917824054 +0000 UTC m=+1772.762537732" Jul 02 13:43:55.674596 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:55.674564 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:43:55.676551 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:55.676531 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:43:58.906835 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:43:58.906806 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd" Jul 02 13:48:55.698040 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:48:55.698008 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:48:55.699934 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:48:55.699897 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:53:55.721138 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:53:55.721099 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:53:55.724474 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:53:55.724453 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:58:17.920523 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:58:17.920486 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 13:58:55.744790 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:58:55.744747 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 13:58:55.750029 ip-10-0-131-243 kubenswrapper[2578]: I0702 13:58:55.750011 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:03:55.773680 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:03:55.773650 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:03:55.778619 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:03:55.778594 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:08:55.797432 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:08:55.797323 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:08:55.809159 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:08:55.809134 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:10:58.101007 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.100909 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh"] Jul 02 14:10:58.104173 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.104155 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" Jul 02 14:10:58.107077 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.107058 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 02 14:10:58.107179 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.107132 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lktlz\"" Jul 02 14:10:58.107245 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.107233 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 02 14:10:58.124854 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.124825 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh"] Jul 02 14:10:58.159724 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.159699 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58612e2d-a020-4726-ae8f-e280a32d52ba-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh\" (UID: \"58612e2d-a020-4726-ae8f-e280a32d52ba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" Jul 02 14:10:58.159819 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.159729 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58612e2d-a020-4726-ae8f-e280a32d52ba-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh\" (UID: \"58612e2d-a020-4726-ae8f-e280a32d52ba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" Jul 02 14:10:58.159819 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.159747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tvsx\" (UniqueName: \"kubernetes.io/projected/58612e2d-a020-4726-ae8f-e280a32d52ba-kube-api-access-9tvsx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh\" (UID: \"58612e2d-a020-4726-ae8f-e280a32d52ba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" Jul 02 14:10:58.260124 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.260098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58612e2d-a020-4726-ae8f-e280a32d52ba-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh\" (UID: \"58612e2d-a020-4726-ae8f-e280a32d52ba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" Jul 02 14:10:58.260238 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.260128 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tvsx\" (UniqueName: \"kubernetes.io/projected/58612e2d-a020-4726-ae8f-e280a32d52ba-kube-api-access-9tvsx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh\" (UID: \"58612e2d-a020-4726-ae8f-e280a32d52ba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" Jul 02 14:10:58.260238 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.260196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58612e2d-a020-4726-ae8f-e280a32d52ba-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh\" (UID: \"58612e2d-a020-4726-ae8f-e280a32d52ba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" Jul 02 14:10:58.260517 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.260494 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58612e2d-a020-4726-ae8f-e280a32d52ba-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh\" (UID: \"58612e2d-a020-4726-ae8f-e280a32d52ba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" Jul 02 14:10:58.260581 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.260534 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58612e2d-a020-4726-ae8f-e280a32d52ba-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh\" (UID: \"58612e2d-a020-4726-ae8f-e280a32d52ba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" Jul 02 14:10:58.270951 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.270903 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tvsx\" (UniqueName: \"kubernetes.io/projected/58612e2d-a020-4726-ae8f-e280a32d52ba-kube-api-access-9tvsx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh\" (UID: \"58612e2d-a020-4726-ae8f-e280a32d52ba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" Jul 02 14:10:58.413425 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.413358 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" Jul 02 14:10:58.543120 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.543015 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh"] Jul 02 14:10:58.545703 ip-10-0-131-243 kubenswrapper[2578]: W0702 14:10:58.545678 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58612e2d_a020_4726_ae8f_e280a32d52ba.slice/crio-8e678297ba5d1dcb2d1936a8f9e620af8515d07effd7bbd822b9a7a8719ca2c9 WatchSource:0}: Error finding container 8e678297ba5d1dcb2d1936a8f9e620af8515d07effd7bbd822b9a7a8719ca2c9: Status 404 returned error can't find the container with id 8e678297ba5d1dcb2d1936a8f9e620af8515d07effd7bbd822b9a7a8719ca2c9 Jul 02 14:10:58.547689 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.547668 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 14:10:58.767454 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:10:58.767378 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" event={"ID":"58612e2d-a020-4726-ae8f-e280a32d52ba","Type":"ContainerStarted","Data":"8e678297ba5d1dcb2d1936a8f9e620af8515d07effd7bbd822b9a7a8719ca2c9"} Jul 02 14:11:06.811793 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:06.811752 2578 generic.go:356] "Generic (PLEG): container finished" podID="58612e2d-a020-4726-ae8f-e280a32d52ba" containerID="6ec7bbd3bac5acd79ab695ac1546d714f4f9764cb0471ace098ace5cb9a847fc" exitCode=0 Jul 02 14:11:06.812171 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:06.811831 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" event={"ID":"58612e2d-a020-4726-ae8f-e280a32d52ba","Type":"ContainerDied","Data":"6ec7bbd3bac5acd79ab695ac1546d714f4f9764cb0471ace098ace5cb9a847fc"} Jul 02 14:11:09.826796 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:09.826762 2578 generic.go:356] "Generic (PLEG): container finished" podID="58612e2d-a020-4726-ae8f-e280a32d52ba" containerID="315aa07a4c388843b7e53eb4ed1c04f36384e6bac8d49d01a182ca3f7b63efcc" exitCode=0 Jul 02 14:11:09.827202 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:09.826845 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" event={"ID":"58612e2d-a020-4726-ae8f-e280a32d52ba","Type":"ContainerDied","Data":"315aa07a4c388843b7e53eb4ed1c04f36384e6bac8d49d01a182ca3f7b63efcc"} Jul 02 14:11:18.879373 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:18.879339 2578 generic.go:356] "Generic (PLEG): container finished" podID="58612e2d-a020-4726-ae8f-e280a32d52ba" containerID="a7bedaefbb7de3fca0ca71e5832baa06308d96661bf073b610078816ece7a5e1" exitCode=0 Jul 02 14:11:18.879712 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:18.879415 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" event={"ID":"58612e2d-a020-4726-ae8f-e280a32d52ba","Type":"ContainerDied","Data":"a7bedaefbb7de3fca0ca71e5832baa06308d96661bf073b610078816ece7a5e1"} Jul 02 14:11:20.050065 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:20.050032 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" Jul 02 14:11:20.158540 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:20.158476 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58612e2d-a020-4726-ae8f-e280a32d52ba-util\") pod \"58612e2d-a020-4726-ae8f-e280a32d52ba\" (UID: \"58612e2d-a020-4726-ae8f-e280a32d52ba\") " Jul 02 14:11:20.158540 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:20.158524 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58612e2d-a020-4726-ae8f-e280a32d52ba-bundle\") pod \"58612e2d-a020-4726-ae8f-e280a32d52ba\" (UID: \"58612e2d-a020-4726-ae8f-e280a32d52ba\") " Jul 02 14:11:20.158690 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:20.158560 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tvsx\" (UniqueName: \"kubernetes.io/projected/58612e2d-a020-4726-ae8f-e280a32d52ba-kube-api-access-9tvsx\") pod \"58612e2d-a020-4726-ae8f-e280a32d52ba\" (UID: \"58612e2d-a020-4726-ae8f-e280a32d52ba\") " Jul 02 14:11:20.159235 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:20.159210 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58612e2d-a020-4726-ae8f-e280a32d52ba-bundle" (OuterVolumeSpecName: "bundle") pod "58612e2d-a020-4726-ae8f-e280a32d52ba" (UID: "58612e2d-a020-4726-ae8f-e280a32d52ba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:11:20.160718 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:20.160699 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58612e2d-a020-4726-ae8f-e280a32d52ba-kube-api-access-9tvsx" (OuterVolumeSpecName: "kube-api-access-9tvsx") pod "58612e2d-a020-4726-ae8f-e280a32d52ba" (UID: "58612e2d-a020-4726-ae8f-e280a32d52ba"). InnerVolumeSpecName "kube-api-access-9tvsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 14:11:20.163253 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:20.163224 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58612e2d-a020-4726-ae8f-e280a32d52ba-util" (OuterVolumeSpecName: "util") pod "58612e2d-a020-4726-ae8f-e280a32d52ba" (UID: "58612e2d-a020-4726-ae8f-e280a32d52ba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:11:20.259450 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:20.259427 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58612e2d-a020-4726-ae8f-e280a32d52ba-util\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:11:20.259450 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:20.259448 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58612e2d-a020-4726-ae8f-e280a32d52ba-bundle\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:11:20.259583 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:20.259456 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tvsx\" (UniqueName: \"kubernetes.io/projected/58612e2d-a020-4726-ae8f-e280a32d52ba-kube-api-access-9tvsx\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:11:20.889096 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:20.889063 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" Jul 02 14:11:20.889096 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:20.889074 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409rrfh" event={"ID":"58612e2d-a020-4726-ae8f-e280a32d52ba","Type":"ContainerDied","Data":"8e678297ba5d1dcb2d1936a8f9e620af8515d07effd7bbd822b9a7a8719ca2c9"} Jul 02 14:11:20.889284 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:20.889115 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e678297ba5d1dcb2d1936a8f9e620af8515d07effd7bbd822b9a7a8719ca2c9" Jul 02 14:11:29.956287 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:29.956253 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7g9zk"] Jul 02 14:11:29.956748 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:29.956537 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58612e2d-a020-4726-ae8f-e280a32d52ba" containerName="extract" Jul 02 14:11:29.956748 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:29.956558 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58612e2d-a020-4726-ae8f-e280a32d52ba" containerName="extract" Jul 02 14:11:29.956748 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:29.956567 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58612e2d-a020-4726-ae8f-e280a32d52ba" containerName="pull" Jul 02 14:11:29.956748 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:29.956573 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58612e2d-a020-4726-ae8f-e280a32d52ba" containerName="pull" Jul 02 14:11:29.956748 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:29.956581 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58612e2d-a020-4726-ae8f-e280a32d52ba" containerName="util" Jul 02 14:11:29.956748 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:29.956586 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58612e2d-a020-4726-ae8f-e280a32d52ba" containerName="util" Jul 02 14:11:29.956748 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:29.956643 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="58612e2d-a020-4726-ae8f-e280a32d52ba" containerName="extract" Jul 02 14:11:29.960692 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:29.960676 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7g9zk" Jul 02 14:11:29.971901 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:29.971872 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7g9zk"] Jul 02 14:11:30.033564 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:30.033537 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c36db6ce-6df8-45df-a5eb-680fff7a8781-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7g9zk\" (UID: \"c36db6ce-6df8-45df-a5eb-680fff7a8781\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7g9zk" Jul 02 14:11:30.033725 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:30.033573 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6smhm\" (UniqueName: \"kubernetes.io/projected/c36db6ce-6df8-45df-a5eb-680fff7a8781-kube-api-access-6smhm\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7g9zk\" (UID: \"c36db6ce-6df8-45df-a5eb-680fff7a8781\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7g9zk" Jul 02 14:11:30.134422 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:30.134394 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c36db6ce-6df8-45df-a5eb-680fff7a8781-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7g9zk\" (UID: \"c36db6ce-6df8-45df-a5eb-680fff7a8781\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7g9zk" Jul 02 14:11:30.134552 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:30.134426 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6smhm\" (UniqueName: \"kubernetes.io/projected/c36db6ce-6df8-45df-a5eb-680fff7a8781-kube-api-access-6smhm\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7g9zk\" (UID: \"c36db6ce-6df8-45df-a5eb-680fff7a8781\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7g9zk" Jul 02 14:11:30.134783 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:30.134760 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c36db6ce-6df8-45df-a5eb-680fff7a8781-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7g9zk\" (UID: \"c36db6ce-6df8-45df-a5eb-680fff7a8781\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7g9zk" Jul 02 14:11:30.143391 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:30.143371 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6smhm\" (UniqueName: \"kubernetes.io/projected/c36db6ce-6df8-45df-a5eb-680fff7a8781-kube-api-access-6smhm\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7g9zk\" (UID: \"c36db6ce-6df8-45df-a5eb-680fff7a8781\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7g9zk" Jul 02 14:11:30.269249 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:30.269185 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7g9zk" Jul 02 14:11:30.402907 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:30.402870 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7g9zk"] Jul 02 14:11:30.406240 ip-10-0-131-243 kubenswrapper[2578]: W0702 14:11:30.406213 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36db6ce_6df8_45df_a5eb_680fff7a8781.slice/crio-50aa07c5df247a04f10b7f3c5a7b08af03d7c433d59acb3cc2028469aa6885a1 WatchSource:0}: Error finding container 50aa07c5df247a04f10b7f3c5a7b08af03d7c433d59acb3cc2028469aa6885a1: Status 404 returned error can't find the container with id 50aa07c5df247a04f10b7f3c5a7b08af03d7c433d59acb3cc2028469aa6885a1 Jul 02 14:11:30.933404 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:30.933369 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7g9zk" event={"ID":"c36db6ce-6df8-45df-a5eb-680fff7a8781","Type":"ContainerStarted","Data":"50aa07c5df247a04f10b7f3c5a7b08af03d7c433d59acb3cc2028469aa6885a1"} Jul 02 14:11:32.944873 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:32.944835 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7g9zk" event={"ID":"c36db6ce-6df8-45df-a5eb-680fff7a8781","Type":"ContainerStarted","Data":"00b56165812ba7b3701d5d9a2e05fb83a432ff7a0b278cb90680d2dadcc58d85"} Jul 02 14:11:32.967071 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:32.967024 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7g9zk" podStartSLOduration=2.300754253 podStartE2EDuration="3.967007767s" totalImagesPullingTime="1.666253514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:11:32.964580251 +0000 UTC m=+3457.809293931" watchObservedRunningTime="2026-07-02 14:11:32.967007767 +0000 UTC m=+3457.811721444" Jul 02 14:11:32.996766 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:32.996736 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz"] Jul 02 14:11:32.996959 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:32.996934 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" podUID="6f24818c-b736-4ed4-a27d-cdf6438f2f2d" containerName="cert-manager-operator" containerID="cri-o://7b97ef8fd45b8592a6123225715ec47f2b165133d007a5b4deb7a5a647300699" gracePeriod=10 Jul 02 14:11:33.248130 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.248105 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" Jul 02 14:11:33.258119 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.258100 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2w9k\" (UniqueName: \"kubernetes.io/projected/6f24818c-b736-4ed4-a27d-cdf6438f2f2d-kube-api-access-x2w9k\") pod \"6f24818c-b736-4ed4-a27d-cdf6438f2f2d\" (UID: \"6f24818c-b736-4ed4-a27d-cdf6438f2f2d\") " Jul 02 14:11:33.258196 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.258152 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f24818c-b736-4ed4-a27d-cdf6438f2f2d-tmp\") pod \"6f24818c-b736-4ed4-a27d-cdf6438f2f2d\" (UID: \"6f24818c-b736-4ed4-a27d-cdf6438f2f2d\") " Jul 02 14:11:33.258524 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.258504 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f24818c-b736-4ed4-a27d-cdf6438f2f2d-tmp" (OuterVolumeSpecName: "tmp") pod "6f24818c-b736-4ed4-a27d-cdf6438f2f2d" (UID: "6f24818c-b736-4ed4-a27d-cdf6438f2f2d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:11:33.260193 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.260170 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f24818c-b736-4ed4-a27d-cdf6438f2f2d-kube-api-access-x2w9k" (OuterVolumeSpecName: "kube-api-access-x2w9k") pod "6f24818c-b736-4ed4-a27d-cdf6438f2f2d" (UID: "6f24818c-b736-4ed4-a27d-cdf6438f2f2d"). InnerVolumeSpecName "kube-api-access-x2w9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 14:11:33.359259 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.359230 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2w9k\" (UniqueName: \"kubernetes.io/projected/6f24818c-b736-4ed4-a27d-cdf6438f2f2d-kube-api-access-x2w9k\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:11:33.359259 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.359257 2578 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f24818c-b736-4ed4-a27d-cdf6438f2f2d-tmp\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:11:33.950810 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.950726 2578 generic.go:356] "Generic (PLEG): container finished" podID="6f24818c-b736-4ed4-a27d-cdf6438f2f2d" containerID="7b97ef8fd45b8592a6123225715ec47f2b165133d007a5b4deb7a5a647300699" exitCode=0 Jul 02 14:11:33.950810 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.950769 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" Jul 02 14:11:33.950810 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.950773 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" event={"ID":"6f24818c-b736-4ed4-a27d-cdf6438f2f2d","Type":"ContainerDied","Data":"7b97ef8fd45b8592a6123225715ec47f2b165133d007a5b4deb7a5a647300699"} Jul 02 14:11:33.951339 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.950827 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz" event={"ID":"6f24818c-b736-4ed4-a27d-cdf6438f2f2d","Type":"ContainerDied","Data":"898b336e6e4e98f6be73a8d56416da1d1b181ac1ec4c695ddb732f20e91c978a"} Jul 02 14:11:33.951339 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.950846 2578 scope.go:117] "RemoveContainer" containerID="7b97ef8fd45b8592a6123225715ec47f2b165133d007a5b4deb7a5a647300699" Jul 02 14:11:33.959190 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.959175 2578 scope.go:117] "RemoveContainer" containerID="7b97ef8fd45b8592a6123225715ec47f2b165133d007a5b4deb7a5a647300699" Jul 02 14:11:33.959429 ip-10-0-131-243 kubenswrapper[2578]: E0702 14:11:33.959412 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b97ef8fd45b8592a6123225715ec47f2b165133d007a5b4deb7a5a647300699\": container with ID starting with 7b97ef8fd45b8592a6123225715ec47f2b165133d007a5b4deb7a5a647300699 not found: ID does not exist" containerID="7b97ef8fd45b8592a6123225715ec47f2b165133d007a5b4deb7a5a647300699" Jul 02 14:11:33.959488 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.959435 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b97ef8fd45b8592a6123225715ec47f2b165133d007a5b4deb7a5a647300699"} err="failed to get container status \"7b97ef8fd45b8592a6123225715ec47f2b165133d007a5b4deb7a5a647300699\": rpc error: code = NotFound desc = could not find container \"7b97ef8fd45b8592a6123225715ec47f2b165133d007a5b4deb7a5a647300699\": container with ID starting with 7b97ef8fd45b8592a6123225715ec47f2b165133d007a5b4deb7a5a647300699 not found: ID does not exist" Jul 02 14:11:33.983250 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.983224 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz"] Jul 02 14:11:33.989930 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:33.989898 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-5b66478c58-drdcz"] Jul 02 14:11:35.625256 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:11:35.625220 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f24818c-b736-4ed4-a27d-cdf6438f2f2d" path="/var/lib/kubelet/pods/6f24818c-b736-4ed4-a27d-cdf6438f2f2d/volumes" Jul 02 14:12:28.645994 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.645958 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gdw64"] Jul 02 14:12:28.646574 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.646384 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f24818c-b736-4ed4-a27d-cdf6438f2f2d" containerName="cert-manager-operator" Jul 02 14:12:28.646574 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.646403 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f24818c-b736-4ed4-a27d-cdf6438f2f2d" containerName="cert-manager-operator" Jul 02 14:12:28.646574 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.646514 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f24818c-b736-4ed4-a27d-cdf6438f2f2d" containerName="cert-manager-operator" Jul 02 14:12:28.648764 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.648742 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xfhtl"] Jul 02 14:12:28.648888 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.648867 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gdw64" Jul 02 14:12:28.650878 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.650858 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4t6gj"] Jul 02 14:12:28.650989 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.650967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xfhtl" Jul 02 14:12:28.652106 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.652086 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 02 14:12:28.652826 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.652808 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4t6gj" Jul 02 14:12:28.654223 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.654198 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-79k77\"" Jul 02 14:12:28.654341 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.654307 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 02 14:12:28.654443 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.654372 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-mq7p2\"" Jul 02 14:12:28.655152 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.655136 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-pb2xc\"" Jul 02 14:12:28.664530 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.664505 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xfhtl"] Jul 02 14:12:28.669170 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.669148 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4t6gj"] Jul 02 14:12:28.674268 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.674239 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gdw64"] Jul 02 14:12:28.801737 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.801706 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a498199-054c-445d-b74c-e515b3854b52-bound-sa-token\") pod \"cert-manager-55b4d954d5-xfhtl\" (UID: \"8a498199-054c-445d-b74c-e515b3854b52\") " pod="cert-manager/cert-manager-55b4d954d5-xfhtl" Jul 02 14:12:28.801880 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.801761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/122e484d-b6fd-4d20-a7e6-719d6d78cf5d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4t6gj\" (UID: \"122e484d-b6fd-4d20-a7e6-719d6d78cf5d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4t6gj" Jul 02 14:12:28.801880 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.801800 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdckb\" (UniqueName: \"kubernetes.io/projected/8d9f8ba2-cecf-4552-abcb-8d3ba5ab6712-kube-api-access-tdckb\") pod \"cert-manager-cainjector-559cdd8c87-gdw64\" (UID: \"8d9f8ba2-cecf-4552-abcb-8d3ba5ab6712\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gdw64" Jul 02 14:12:28.801880 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.801832 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k58fp\" (UniqueName: \"kubernetes.io/projected/8a498199-054c-445d-b74c-e515b3854b52-kube-api-access-k58fp\") pod \"cert-manager-55b4d954d5-xfhtl\" (UID: \"8a498199-054c-445d-b74c-e515b3854b52\") " pod="cert-manager/cert-manager-55b4d954d5-xfhtl" Jul 02 14:12:28.802022 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.801889 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqskg\" (UniqueName: \"kubernetes.io/projected/122e484d-b6fd-4d20-a7e6-719d6d78cf5d-kube-api-access-wqskg\") pod \"cert-manager-webhook-6486ffd997-4t6gj\" (UID: \"122e484d-b6fd-4d20-a7e6-719d6d78cf5d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4t6gj" Jul 02 14:12:28.802022 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.801905 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d9f8ba2-cecf-4552-abcb-8d3ba5ab6712-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gdw64\" (UID: \"8d9f8ba2-cecf-4552-abcb-8d3ba5ab6712\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gdw64" Jul 02 14:12:28.903062 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.902990 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdckb\" (UniqueName: \"kubernetes.io/projected/8d9f8ba2-cecf-4552-abcb-8d3ba5ab6712-kube-api-access-tdckb\") pod \"cert-manager-cainjector-559cdd8c87-gdw64\" (UID: \"8d9f8ba2-cecf-4552-abcb-8d3ba5ab6712\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gdw64" Jul 02 14:12:28.903062 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.903025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k58fp\" (UniqueName: \"kubernetes.io/projected/8a498199-054c-445d-b74c-e515b3854b52-kube-api-access-k58fp\") pod \"cert-manager-55b4d954d5-xfhtl\" (UID: \"8a498199-054c-445d-b74c-e515b3854b52\") " pod="cert-manager/cert-manager-55b4d954d5-xfhtl" Jul 02 14:12:28.903062 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.903059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqskg\" (UniqueName: \"kubernetes.io/projected/122e484d-b6fd-4d20-a7e6-719d6d78cf5d-kube-api-access-wqskg\") pod \"cert-manager-webhook-6486ffd997-4t6gj\" (UID: \"122e484d-b6fd-4d20-a7e6-719d6d78cf5d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4t6gj" Jul 02 14:12:28.903307 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.903075 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d9f8ba2-cecf-4552-abcb-8d3ba5ab6712-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gdw64\" (UID: \"8d9f8ba2-cecf-4552-abcb-8d3ba5ab6712\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gdw64" Jul 02 14:12:28.903307 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.903101 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a498199-054c-445d-b74c-e515b3854b52-bound-sa-token\") pod \"cert-manager-55b4d954d5-xfhtl\" (UID: \"8a498199-054c-445d-b74c-e515b3854b52\") " pod="cert-manager/cert-manager-55b4d954d5-xfhtl" Jul 02 14:12:28.903307 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.903145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/122e484d-b6fd-4d20-a7e6-719d6d78cf5d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4t6gj\" (UID: \"122e484d-b6fd-4d20-a7e6-719d6d78cf5d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4t6gj" Jul 02 14:12:28.916845 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.916824 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k58fp\" (UniqueName: \"kubernetes.io/projected/8a498199-054c-445d-b74c-e515b3854b52-kube-api-access-k58fp\") pod \"cert-manager-55b4d954d5-xfhtl\" (UID: \"8a498199-054c-445d-b74c-e515b3854b52\") " pod="cert-manager/cert-manager-55b4d954d5-xfhtl" Jul 02 14:12:28.919136 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.919113 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqskg\" (UniqueName: \"kubernetes.io/projected/122e484d-b6fd-4d20-a7e6-719d6d78cf5d-kube-api-access-wqskg\") pod \"cert-manager-webhook-6486ffd997-4t6gj\" (UID: \"122e484d-b6fd-4d20-a7e6-719d6d78cf5d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4t6gj" Jul 02 14:12:28.921134 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.921115 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a498199-054c-445d-b74c-e515b3854b52-bound-sa-token\") pod \"cert-manager-55b4d954d5-xfhtl\" (UID: \"8a498199-054c-445d-b74c-e515b3854b52\") " pod="cert-manager/cert-manager-55b4d954d5-xfhtl" Jul 02 14:12:28.922718 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.922694 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdckb\" (UniqueName: \"kubernetes.io/projected/8d9f8ba2-cecf-4552-abcb-8d3ba5ab6712-kube-api-access-tdckb\") pod \"cert-manager-cainjector-559cdd8c87-gdw64\" (UID: \"8d9f8ba2-cecf-4552-abcb-8d3ba5ab6712\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gdw64" Jul 02 14:12:28.923994 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.923970 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/122e484d-b6fd-4d20-a7e6-719d6d78cf5d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4t6gj\" (UID: \"122e484d-b6fd-4d20-a7e6-719d6d78cf5d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4t6gj" Jul 02 14:12:28.924489 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.924468 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d9f8ba2-cecf-4552-abcb-8d3ba5ab6712-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gdw64\" (UID: \"8d9f8ba2-cecf-4552-abcb-8d3ba5ab6712\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gdw64" Jul 02 14:12:28.959309 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.959286 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gdw64" Jul 02 14:12:28.965754 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.965735 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xfhtl" Jul 02 14:12:28.977541 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:28.977522 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4t6gj" Jul 02 14:12:29.117298 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:29.117259 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gdw64"] Jul 02 14:12:29.121197 ip-10-0-131-243 kubenswrapper[2578]: W0702 14:12:29.121171 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d9f8ba2_cecf_4552_abcb_8d3ba5ab6712.slice/crio-0ba5d1fbadf825fa8c6bd7d25c7ac8f540dbafc9ad2bfd2ac92dcf61c674aa37 WatchSource:0}: Error finding container 0ba5d1fbadf825fa8c6bd7d25c7ac8f540dbafc9ad2bfd2ac92dcf61c674aa37: Status 404 returned error can't find the container with id 0ba5d1fbadf825fa8c6bd7d25c7ac8f540dbafc9ad2bfd2ac92dcf61c674aa37 Jul 02 14:12:29.192169 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:29.192102 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gdw64" event={"ID":"8d9f8ba2-cecf-4552-abcb-8d3ba5ab6712","Type":"ContainerStarted","Data":"0ba5d1fbadf825fa8c6bd7d25c7ac8f540dbafc9ad2bfd2ac92dcf61c674aa37"} Jul 02 14:12:29.353736 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:29.353698 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4t6gj"] Jul 02 14:12:29.355803 ip-10-0-131-243 kubenswrapper[2578]: W0702 14:12:29.355774 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod122e484d_b6fd_4d20_a7e6_719d6d78cf5d.slice/crio-031be1080d7e6802bf82302d2237e7cec30652bb5223ec1375e3b64b21789cc3 WatchSource:0}: Error finding container 031be1080d7e6802bf82302d2237e7cec30652bb5223ec1375e3b64b21789cc3: Status 404 returned error can't find the container with id 031be1080d7e6802bf82302d2237e7cec30652bb5223ec1375e3b64b21789cc3 Jul 02 14:12:29.360096 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:29.360047 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xfhtl"] Jul 02 14:12:29.361844 ip-10-0-131-243 kubenswrapper[2578]: W0702 14:12:29.361821 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a498199_054c_445d_b74c_e515b3854b52.slice/crio-7d72224e5373a34e7d4d7c7162de8dd51ebe8c95067ccb8f24f72b34ef427ec5 WatchSource:0}: Error finding container 7d72224e5373a34e7d4d7c7162de8dd51ebe8c95067ccb8f24f72b34ef427ec5: Status 404 returned error can't find the container with id 7d72224e5373a34e7d4d7c7162de8dd51ebe8c95067ccb8f24f72b34ef427ec5 Jul 02 14:12:30.199591 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:30.199533 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xfhtl" event={"ID":"8a498199-054c-445d-b74c-e515b3854b52","Type":"ContainerStarted","Data":"7d72224e5373a34e7d4d7c7162de8dd51ebe8c95067ccb8f24f72b34ef427ec5"} Jul 02 14:12:30.201271 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:30.201224 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4t6gj" event={"ID":"122e484d-b6fd-4d20-a7e6-719d6d78cf5d","Type":"ContainerStarted","Data":"031be1080d7e6802bf82302d2237e7cec30652bb5223ec1375e3b64b21789cc3"} Jul 02 14:12:33.220262 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:33.220227 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4t6gj" event={"ID":"122e484d-b6fd-4d20-a7e6-719d6d78cf5d","Type":"ContainerStarted","Data":"c5f541519ba612243d717c187aeb00bf0414e668a3ec78140d7a80c88dbdb498"} Jul 02 14:12:33.220766 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:33.220319 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4t6gj" Jul 02 14:12:33.221759 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:33.221725 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gdw64" event={"ID":"8d9f8ba2-cecf-4552-abcb-8d3ba5ab6712","Type":"ContainerStarted","Data":"f153c44b3abe46a0b37d65d47c56c1fc3735f80a18d37caf6bb90ebd1159aaa8"} Jul 02 14:12:33.222963 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:33.222940 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xfhtl" event={"ID":"8a498199-054c-445d-b74c-e515b3854b52","Type":"ContainerStarted","Data":"1c73d8e040540d8cae7277a6b1b9f978f1f17b61b54ff268e79a447a7e3c64f1"} Jul 02 14:12:33.240725 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:33.240682 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-4t6gj" podStartSLOduration=2.3057920579999998 podStartE2EDuration="5.24066907s" totalImagesPullingTime="2.934877012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:12:33.239188098 +0000 UTC m=+3518.083901778" watchObservedRunningTime="2026-07-02 14:12:33.24066907 +0000 UTC m=+3518.085382744" Jul 02 14:12:33.257214 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:33.257166 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xfhtl" podStartSLOduration=2.318183575 podStartE2EDuration="5.257152503s" totalImagesPullingTime="2.938968928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:12:33.256455739 +0000 UTC m=+3518.101169417" watchObservedRunningTime="2026-07-02 14:12:33.257152503 +0000 UTC m=+3518.101866181" Jul 02 14:12:33.274599 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:33.274558 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gdw64" podStartSLOduration=2.104876421 podStartE2EDuration="5.274543573s" totalImagesPullingTime="3.169667152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:12:33.273655073 +0000 UTC m=+3518.118368753" watchObservedRunningTime="2026-07-02 14:12:33.274543573 +0000 UTC m=+3518.119257250" Jul 02 14:12:39.230330 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:12:39.230298 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4t6gj" Jul 02 14:13:55.826363 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:13:55.826333 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:13:55.835494 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:13:55.835475 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:16:34.224949 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.224896 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5"] Jul 02 14:16:34.227481 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.227464 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:16:34.232029 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.232009 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-tbcnn\"" Jul 02 14:16:34.233144 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.233127 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\"" Jul 02 14:16:34.248595 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.248562 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5\" (UID: \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:16:34.248762 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.248635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm27r\" (UniqueName: \"kubernetes.io/projected/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-kube-api-access-cm27r\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5\" (UID: \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:16:34.248762 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.248736 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5\" (UID: \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:16:34.250379 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.250354 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5"] Jul 02 14:16:34.349746 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.349708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5\" (UID: \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:16:34.349977 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.349761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm27r\" (UniqueName: \"kubernetes.io/projected/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-kube-api-access-cm27r\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5\" (UID: \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:16:34.349977 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.349805 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5\" (UID: \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:16:34.350201 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.350177 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5\" (UID: \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:16:34.352248 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.352223 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5\" (UID: \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:16:34.359981 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.359953 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm27r\" (UniqueName: \"kubernetes.io/projected/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-kube-api-access-cm27r\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5\" (UID: \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:16:34.537621 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.537525 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:16:34.672908 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:34.672718 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5"] Jul 02 14:16:34.675627 ip-10-0-131-243 kubenswrapper[2578]: W0702 14:16:34.675593 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9c9543_c3d8_4d82_b4c2_b9a13765ed00.slice/crio-26b494d0690054b6a4251e49a6fbdbb2a5e46bd351dd2ac98bb6f670ca5162c0 WatchSource:0}: Error finding container 26b494d0690054b6a4251e49a6fbdbb2a5e46bd351dd2ac98bb6f670ca5162c0: Status 404 returned error can't find the container with id 26b494d0690054b6a4251e49a6fbdbb2a5e46bd351dd2ac98bb6f670ca5162c0 Jul 02 14:16:35.319818 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:35.319771 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" event={"ID":"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00","Type":"ContainerStarted","Data":"f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2"} Jul 02 14:16:35.320365 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:35.319875 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:16:35.320365 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:35.319897 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" event={"ID":"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00","Type":"ContainerStarted","Data":"26b494d0690054b6a4251e49a6fbdbb2a5e46bd351dd2ac98bb6f670ca5162c0"} Jul 02 14:16:35.341783 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:16:35.341718 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" podStartSLOduration=1.341698913 podStartE2EDuration="1.341698913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:35.341063057 +0000 UTC m=+3760.185776757" watchObservedRunningTime="2026-07-02 14:16:35.341698913 +0000 UTC m=+3760.186412592" Jul 02 14:17:06.326687 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:17:06.326609 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:18:12.011950 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.011901 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5"] Jul 02 14:18:12.022498 ip-10-0-131-243 kubenswrapper[2578]: E0702 14:18:12.022461 2578 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 02 14:18:12.022498 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 14:18:12.022498 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 14:18:12.022498 ip-10-0-131-243 kubenswrapper[2578]: > containerID="f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2" cmd=["/bin/sleep","15"] Jul 02 14:18:12.022752 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.022519 2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" message="" err=< Jul 02 14:18:12.022752 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 14:18:12.022752 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 14:18:12.022752 ip-10-0-131-243 kubenswrapper[2578]: > Jul 02 14:18:12.022752 ip-10-0-131-243 kubenswrapper[2578]: E0702 14:18:12.022537 2578 kuberuntime_container.go:754] "PreStop hook failed" err=< Jul 02 14:18:12.022752 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 14:18:12.022752 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 14:18:12.022752 ip-10-0-131-243 kubenswrapper[2578]: > pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" podUID="1c9c9543-c3d8-4d82-b4c2-b9a13765ed00" containerName="main" containerID="cri-o://f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2" Jul 02 14:18:12.022752 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.022583 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" podUID="1c9c9543-c3d8-4d82-b4c2-b9a13765ed00" containerName="main" containerID="cri-o://f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2" gracePeriod=60 Jul 02 14:18:12.277283 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.277222 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:18:12.391689 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.391654 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm27r\" (UniqueName: \"kubernetes.io/projected/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-kube-api-access-cm27r\") pod \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\" (UID: \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\") " Jul 02 14:18:12.391862 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.391723 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-tls-certs\") pod \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\" (UID: \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\") " Jul 02 14:18:12.391862 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.391769 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-tokenizer-uds\") pod \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\" (UID: \"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00\") " Jul 02 14:18:12.392069 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.392048 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1c9c9543-c3d8-4d82-b4c2-b9a13765ed00" (UID: "1c9c9543-c3d8-4d82-b4c2-b9a13765ed00"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:18:12.393861 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.393833 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-kube-api-access-cm27r" (OuterVolumeSpecName: "kube-api-access-cm27r") pod "1c9c9543-c3d8-4d82-b4c2-b9a13765ed00" (UID: "1c9c9543-c3d8-4d82-b4c2-b9a13765ed00"). InnerVolumeSpecName "kube-api-access-cm27r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 14:18:12.394010 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.393871 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c9c9543-c3d8-4d82-b4c2-b9a13765ed00" (UID: "1c9c9543-c3d8-4d82-b4c2-b9a13765ed00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 14:18:12.493036 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.493006 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-tls-certs\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:18:12.493036 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.493034 2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-tokenizer-uds\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:18:12.493036 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.493044 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm27r\" (UniqueName: \"kubernetes.io/projected/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00-kube-api-access-cm27r\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:18:12.758276 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.758244 2578 generic.go:356] "Generic (PLEG): container finished" podID="1c9c9543-c3d8-4d82-b4c2-b9a13765ed00" containerID="f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2" exitCode=0 Jul 02 14:18:12.758466 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.758292 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" Jul 02 14:18:12.758466 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.758329 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" event={"ID":"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00","Type":"ContainerDied","Data":"f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2"} Jul 02 14:18:12.758466 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.758384 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5" event={"ID":"1c9c9543-c3d8-4d82-b4c2-b9a13765ed00","Type":"ContainerDied","Data":"26b494d0690054b6a4251e49a6fbdbb2a5e46bd351dd2ac98bb6f670ca5162c0"} Jul 02 14:18:12.758466 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.758405 2578 scope.go:117] "RemoveContainer" containerID="f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2" Jul 02 14:18:12.767966 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.767949 2578 scope.go:117] "RemoveContainer" containerID="f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2" Jul 02 14:18:12.768240 ip-10-0-131-243 kubenswrapper[2578]: E0702 14:18:12.768221 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2\": container with ID starting with f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2 not found: ID does not exist" containerID="f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2" Jul 02 14:18:12.768303 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.768250 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2"} err="failed to get container status \"f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2\": rpc error: code = NotFound desc = could not find container \"f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2\": container with ID starting with f98208323028dd34891b6dee8941184d69d047bb54cd0b2696dd83d4dd10c2b2 not found: ID does not exist" Jul 02 14:18:12.783927 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.783891 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5"] Jul 02 14:18:12.794511 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:12.794484 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5"] Jul 02 14:18:13.625571 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:13.625535 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c9c9543-c3d8-4d82-b4c2-b9a13765ed00" path="/var/lib/kubelet/pods/1c9c9543-c3d8-4d82-b4c2-b9a13765ed00/volumes" Jul 02 14:18:55.852608 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:55.852500 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:18:55.861111 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:18:55.861089 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:19:08.403874 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.403791 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k"] Jul 02 14:19:08.404287 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.404269 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c9c9543-c3d8-4d82-b4c2-b9a13765ed00" containerName="main" Jul 02 14:19:08.404287 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.404285 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c9c9543-c3d8-4d82-b4c2-b9a13765ed00" containerName="main" Jul 02 14:19:08.404399 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.404387 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c9c9543-c3d8-4d82-b4c2-b9a13765ed00" containerName="main" Jul 02 14:19:08.408462 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.408444 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:19:08.412376 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.412356 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-4gnvm\"" Jul 02 14:19:08.412489 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.412439 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\"" Jul 02 14:19:08.417840 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.417812 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k"] Jul 02 14:19:08.481280 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.481251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b6a6deb6-c6f1-4968-b08d-42e533ea421e-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k\" (UID: \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:19:08.481436 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.481285 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wdrh\" (UniqueName: \"kubernetes.io/projected/b6a6deb6-c6f1-4968-b08d-42e533ea421e-kube-api-access-7wdrh\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k\" (UID: \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:19:08.481436 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.481312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a6deb6-c6f1-4968-b08d-42e533ea421e-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k\" (UID: \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:19:08.582092 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.582058 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b6a6deb6-c6f1-4968-b08d-42e533ea421e-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k\" (UID: \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:19:08.582262 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.582095 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wdrh\" (UniqueName: \"kubernetes.io/projected/b6a6deb6-c6f1-4968-b08d-42e533ea421e-kube-api-access-7wdrh\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k\" (UID: \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:19:08.582262 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.582123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a6deb6-c6f1-4968-b08d-42e533ea421e-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k\" (UID: \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:19:08.582503 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.582484 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b6a6deb6-c6f1-4968-b08d-42e533ea421e-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k\" (UID: \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:19:08.584552 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.584526 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a6deb6-c6f1-4968-b08d-42e533ea421e-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k\" (UID: \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:19:08.593660 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.593636 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wdrh\" (UniqueName: \"kubernetes.io/projected/b6a6deb6-c6f1-4968-b08d-42e533ea421e-kube-api-access-7wdrh\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k\" (UID: \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:19:08.719193 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.719085 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:19:08.853610 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:08.853580 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k"] Jul 02 14:19:08.855307 ip-10-0-131-243 kubenswrapper[2578]: W0702 14:19:08.855282 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a6deb6_c6f1_4968_b08d_42e533ea421e.slice/crio-65f062da321d1a3fe2c98313d5214c651f5fb276133db9aaf279e3a4445e7192 WatchSource:0}: Error finding container 65f062da321d1a3fe2c98313d5214c651f5fb276133db9aaf279e3a4445e7192: Status 404 returned error can't find the container with id 65f062da321d1a3fe2c98313d5214c651f5fb276133db9aaf279e3a4445e7192 Jul 02 14:19:09.018468 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:09.018367 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" event={"ID":"b6a6deb6-c6f1-4968-b08d-42e533ea421e","Type":"ContainerStarted","Data":"72b7a09baa5bd429951ffc39c2f70cdbbb3ffbdd976f54072c96327c4b06b158"} Jul 02 14:19:09.018468 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:09.018428 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" event={"ID":"b6a6deb6-c6f1-4968-b08d-42e533ea421e","Type":"ContainerStarted","Data":"65f062da321d1a3fe2c98313d5214c651f5fb276133db9aaf279e3a4445e7192"} Jul 02 14:19:09.018700 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:09.018471 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:19:09.038549 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:09.038501 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" podStartSLOduration=1.038484104 podStartE2EDuration="1.038484104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:19:09.037015779 +0000 UTC m=+3913.881729463" watchObservedRunningTime="2026-07-02 14:19:09.038484104 +0000 UTC m=+3913.883197782" Jul 02 14:19:40.025079 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:19:40.025047 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:20:23.190947 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.190897 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k"] Jul 02 14:20:23.200126 ip-10-0-131-243 kubenswrapper[2578]: E0702 14:20:23.200082 2578 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 02 14:20:23.200126 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 14:20:23.200126 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 14:20:23.200126 ip-10-0-131-243 kubenswrapper[2578]: > containerID="72b7a09baa5bd429951ffc39c2f70cdbbb3ffbdd976f54072c96327c4b06b158" cmd=["/bin/sleep","15"] Jul 02 14:20:23.200387 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.200147 2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" message="" err=< Jul 02 14:20:23.200387 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 14:20:23.200387 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 14:20:23.200387 ip-10-0-131-243 kubenswrapper[2578]: > Jul 02 14:20:23.200387 ip-10-0-131-243 kubenswrapper[2578]: E0702 14:20:23.200163 2578 kuberuntime_container.go:754] "PreStop hook failed" err=< Jul 02 14:20:23.200387 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 14:20:23.200387 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 14:20:23.200387 ip-10-0-131-243 kubenswrapper[2578]: > pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" podUID="b6a6deb6-c6f1-4968-b08d-42e533ea421e" containerName="main" containerID="cri-o://72b7a09baa5bd429951ffc39c2f70cdbbb3ffbdd976f54072c96327c4b06b158" Jul 02 14:20:23.200387 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.200206 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" podUID="b6a6deb6-c6f1-4968-b08d-42e533ea421e" containerName="main" containerID="cri-o://72b7a09baa5bd429951ffc39c2f70cdbbb3ffbdd976f54072c96327c4b06b158" gracePeriod=60 Jul 02 14:20:23.358887 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.358852 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6a6deb6-c6f1-4968-b08d-42e533ea421e" containerID="72b7a09baa5bd429951ffc39c2f70cdbbb3ffbdd976f54072c96327c4b06b158" exitCode=0 Jul 02 14:20:23.358887 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.358893 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" event={"ID":"b6a6deb6-c6f1-4968-b08d-42e533ea421e","Type":"ContainerDied","Data":"72b7a09baa5bd429951ffc39c2f70cdbbb3ffbdd976f54072c96327c4b06b158"} Jul 02 14:20:23.448964 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.448880 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:20:23.500176 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.500145 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b6a6deb6-c6f1-4968-b08d-42e533ea421e-tokenizer-uds\") pod \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\" (UID: \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\") " Jul 02 14:20:23.500358 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.500225 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wdrh\" (UniqueName: \"kubernetes.io/projected/b6a6deb6-c6f1-4968-b08d-42e533ea421e-kube-api-access-7wdrh\") pod \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\" (UID: \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\") " Jul 02 14:20:23.500358 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.500277 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a6deb6-c6f1-4968-b08d-42e533ea421e-tls-certs\") pod \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\" (UID: \"b6a6deb6-c6f1-4968-b08d-42e533ea421e\") " Jul 02 14:20:23.500473 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.500426 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a6deb6-c6f1-4968-b08d-42e533ea421e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b6a6deb6-c6f1-4968-b08d-42e533ea421e" (UID: "b6a6deb6-c6f1-4968-b08d-42e533ea421e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:20:23.500516 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.500507 2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b6a6deb6-c6f1-4968-b08d-42e533ea421e-tokenizer-uds\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:20:23.502372 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.502346 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a6deb6-c6f1-4968-b08d-42e533ea421e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6a6deb6-c6f1-4968-b08d-42e533ea421e" (UID: "b6a6deb6-c6f1-4968-b08d-42e533ea421e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 14:20:23.502468 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.502395 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a6deb6-c6f1-4968-b08d-42e533ea421e-kube-api-access-7wdrh" (OuterVolumeSpecName: "kube-api-access-7wdrh") pod "b6a6deb6-c6f1-4968-b08d-42e533ea421e" (UID: "b6a6deb6-c6f1-4968-b08d-42e533ea421e"). InnerVolumeSpecName "kube-api-access-7wdrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 14:20:23.601759 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.601722 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a6deb6-c6f1-4968-b08d-42e533ea421e-tls-certs\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:20:23.601759 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:23.601752 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wdrh\" (UniqueName: \"kubernetes.io/projected/b6a6deb6-c6f1-4968-b08d-42e533ea421e-kube-api-access-7wdrh\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:20:24.365082 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:24.365050 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" event={"ID":"b6a6deb6-c6f1-4968-b08d-42e533ea421e","Type":"ContainerDied","Data":"65f062da321d1a3fe2c98313d5214c651f5fb276133db9aaf279e3a4445e7192"} Jul 02 14:20:24.365518 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:24.365110 2578 scope.go:117] "RemoveContainer" containerID="72b7a09baa5bd429951ffc39c2f70cdbbb3ffbdd976f54072c96327c4b06b158" Jul 02 14:20:24.365518 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:24.365061 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k" Jul 02 14:20:24.382779 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:24.382752 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k"] Jul 02 14:20:24.388974 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:24.388950 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k"] Jul 02 14:20:25.629799 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:20:25.629757 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6a6deb6-c6f1-4968-b08d-42e533ea421e" path="/var/lib/kubelet/pods/b6a6deb6-c6f1-4968-b08d-42e533ea421e/volumes" Jul 02 14:23:55.878692 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:55.878578 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:23:55.891618 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:55.891598 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:23:58.130291 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.130248 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq"] Jul 02 14:23:58.130666 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.130604 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a6deb6-c6f1-4968-b08d-42e533ea421e" containerName="main" Jul 02 14:23:58.130666 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.130620 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a6deb6-c6f1-4968-b08d-42e533ea421e" containerName="main" Jul 02 14:23:58.130903 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.130680 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a6deb6-c6f1-4968-b08d-42e533ea421e" containerName="main" Jul 02 14:23:58.133593 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.133576 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" Jul 02 14:23:58.137184 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.137161 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\"" Jul 02 14:23:58.137278 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.137183 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-c54fj\"" Jul 02 14:23:58.150898 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.150874 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq"] Jul 02 14:23:58.176064 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.176039 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d4ff04-0aaa-4989-9d49-725e5213cbf7-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq\" (UID: \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" Jul 02 14:23:58.176174 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.176136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/24d4ff04-0aaa-4989-9d49-725e5213cbf7-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq\" (UID: \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" Jul 02 14:23:58.176223 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.176192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv2n2\" (UniqueName: \"kubernetes.io/projected/24d4ff04-0aaa-4989-9d49-725e5213cbf7-kube-api-access-hv2n2\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq\" (UID: \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" Jul 02 14:23:58.276855 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.276823 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/24d4ff04-0aaa-4989-9d49-725e5213cbf7-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq\" (UID: \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" Jul 02 14:23:58.277086 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.276879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv2n2\" (UniqueName: \"kubernetes.io/projected/24d4ff04-0aaa-4989-9d49-725e5213cbf7-kube-api-access-hv2n2\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq\" (UID: \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" Jul 02 14:23:58.277086 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.277006 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d4ff04-0aaa-4989-9d49-725e5213cbf7-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq\" (UID: \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" Jul 02 14:23:58.277317 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.277292 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/24d4ff04-0aaa-4989-9d49-725e5213cbf7-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq\" (UID: \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" Jul 02 14:23:58.279418 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.279392 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d4ff04-0aaa-4989-9d49-725e5213cbf7-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq\" (UID: \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" Jul 02 14:23:58.286199 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.286173 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv2n2\" (UniqueName: \"kubernetes.io/projected/24d4ff04-0aaa-4989-9d49-725e5213cbf7-kube-api-access-hv2n2\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq\" (UID: \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" Jul 02 14:23:58.442250 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.442173 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" Jul 02 14:23:58.588129 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:58.588104 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq"] Jul 02 14:23:58.590667 ip-10-0-131-243 kubenswrapper[2578]: W0702 14:23:58.590640 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d4ff04_0aaa_4989_9d49_725e5213cbf7.slice/crio-5afbce1be8878b93c7a275acf3e91b85ab25c10e36a46e22b4f8d293da17bebd WatchSource:0}: Error finding container 5afbce1be8878b93c7a275acf3e91b85ab25c10e36a46e22b4f8d293da17bebd: Status 404 returned error can't find the container with id 5afbce1be8878b93c7a275acf3e91b85ab25c10e36a46e22b4f8d293da17bebd Jul 02 14:23:59.358715 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:59.358682 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" event={"ID":"24d4ff04-0aaa-4989-9d49-725e5213cbf7","Type":"ContainerStarted","Data":"66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8"} Jul 02 14:23:59.359095 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:59.358730 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" event={"ID":"24d4ff04-0aaa-4989-9d49-725e5213cbf7","Type":"ContainerStarted","Data":"5afbce1be8878b93c7a275acf3e91b85ab25c10e36a46e22b4f8d293da17bebd"} Jul 02 14:23:59.359095 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:59.358804 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" Jul 02 14:23:59.380770 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:23:59.380724 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" podStartSLOduration=1.380710225 podStartE2EDuration="1.380710225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:23:59.378781841 +0000 UTC m=+4204.223495520" watchObservedRunningTime="2026-07-02 14:23:59.380710225 +0000 UTC m=+4204.225423903" Jul 02 14:24:30.365815 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:30.365777 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" podUID="24d4ff04-0aaa-4989-9d49-725e5213cbf7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 02 14:24:40.365698 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:40.365656 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" podUID="24d4ff04-0aaa-4989-9d49-725e5213cbf7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 02 14:24:50.366310 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:50.366270 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" podUID="24d4ff04-0aaa-4989-9d49-725e5213cbf7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 02 14:24:53.041326 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.041275 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq"] Jul 02 14:24:53.044498 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.044460 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" podUID="24d4ff04-0aaa-4989-9d49-725e5213cbf7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Jul 02 14:24:53.053540 ip-10-0-131-243 kubenswrapper[2578]: E0702 14:24:53.053505 2578 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 02 14:24:53.053540 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 14:24:53.053540 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 14:24:53.053540 ip-10-0-131-243 kubenswrapper[2578]: > containerID="66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8" cmd=["/bin/sleep","15"] Jul 02 14:24:53.053840 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.053560 2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" message="" err=< Jul 02 14:24:53.053840 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 14:24:53.053840 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 14:24:53.053840 ip-10-0-131-243 kubenswrapper[2578]: > Jul 02 14:24:53.053840 ip-10-0-131-243 kubenswrapper[2578]: E0702 14:24:53.053579 2578 kuberuntime_container.go:754] "PreStop hook failed" err=< Jul 02 14:24:53.053840 ip-10-0-131-243 kubenswrapper[2578]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 02 14:24:53.053840 ip-10-0-131-243 kubenswrapper[2578]: , stdout: , stderr: , exit code -1 Jul 02 14:24:53.053840 ip-10-0-131-243 kubenswrapper[2578]: > pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" podUID="24d4ff04-0aaa-4989-9d49-725e5213cbf7" containerName="main" containerID="cri-o://66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8" Jul 02 14:24:53.053840 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.053619 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" podUID="24d4ff04-0aaa-4989-9d49-725e5213cbf7" containerName="main" containerID="cri-o://66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8" gracePeriod=60 Jul 02 14:24:53.311984 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.311952 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" Jul 02 14:24:53.454966 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.454909 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv2n2\" (UniqueName: \"kubernetes.io/projected/24d4ff04-0aaa-4989-9d49-725e5213cbf7-kube-api-access-hv2n2\") pod \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\" (UID: \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\") " Jul 02 14:24:53.455167 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.455019 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/24d4ff04-0aaa-4989-9d49-725e5213cbf7-tokenizer-uds\") pod \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\" (UID: \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\") " Jul 02 14:24:53.455167 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.455065 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d4ff04-0aaa-4989-9d49-725e5213cbf7-tls-certs\") pod \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\" (UID: \"24d4ff04-0aaa-4989-9d49-725e5213cbf7\") " Jul 02 14:24:53.455316 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.455291 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d4ff04-0aaa-4989-9d49-725e5213cbf7-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "24d4ff04-0aaa-4989-9d49-725e5213cbf7" (UID: "24d4ff04-0aaa-4989-9d49-725e5213cbf7"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:24:53.457121 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.457096 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d4ff04-0aaa-4989-9d49-725e5213cbf7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24d4ff04-0aaa-4989-9d49-725e5213cbf7" (UID: "24d4ff04-0aaa-4989-9d49-725e5213cbf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 14:24:53.457248 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.457146 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d4ff04-0aaa-4989-9d49-725e5213cbf7-kube-api-access-hv2n2" (OuterVolumeSpecName: "kube-api-access-hv2n2") pod "24d4ff04-0aaa-4989-9d49-725e5213cbf7" (UID: "24d4ff04-0aaa-4989-9d49-725e5213cbf7"). InnerVolumeSpecName "kube-api-access-hv2n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 14:24:53.556718 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.556629 2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/24d4ff04-0aaa-4989-9d49-725e5213cbf7-tokenizer-uds\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:24:53.556718 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.556658 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d4ff04-0aaa-4989-9d49-725e5213cbf7-tls-certs\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:24:53.556718 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.556668 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv2n2\" (UniqueName: \"kubernetes.io/projected/24d4ff04-0aaa-4989-9d49-725e5213cbf7-kube-api-access-hv2n2\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:24:53.616954 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.616904 2578 generic.go:356] "Generic (PLEG): container finished" podID="24d4ff04-0aaa-4989-9d49-725e5213cbf7" containerID="66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8" exitCode=0 Jul 02 14:24:53.617149 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.616956 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" event={"ID":"24d4ff04-0aaa-4989-9d49-725e5213cbf7","Type":"ContainerDied","Data":"66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8"} Jul 02 14:24:53.617149 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.617021 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" event={"ID":"24d4ff04-0aaa-4989-9d49-725e5213cbf7","Type":"ContainerDied","Data":"5afbce1be8878b93c7a275acf3e91b85ab25c10e36a46e22b4f8d293da17bebd"} Jul 02 14:24:53.617149 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.617049 2578 scope.go:117] "RemoveContainer" containerID="66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8" Jul 02 14:24:53.617149 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.616967 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq" Jul 02 14:24:53.627064 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.626958 2578 scope.go:117] "RemoveContainer" containerID="66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8" Jul 02 14:24:53.627237 ip-10-0-131-243 kubenswrapper[2578]: E0702 14:24:53.627212 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8\": container with ID starting with 66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8 not found: ID does not exist" containerID="66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8" Jul 02 14:24:53.627294 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.627245 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8"} err="failed to get container status \"66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8\": rpc error: code = NotFound desc = could not find container \"66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8\": container with ID starting with 66085f108e34dd850273eaf20c15774a9794a9ba566244c1809fd242717e8eb8 not found: ID does not exist" Jul 02 14:24:53.640650 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.640622 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq"] Jul 02 14:24:53.645465 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:53.645444 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq"] Jul 02 14:24:55.625019 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:24:55.624984 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24d4ff04-0aaa-4989-9d49-725e5213cbf7" path="/var/lib/kubelet/pods/24d4ff04-0aaa-4989-9d49-725e5213cbf7/volumes" Jul 02 14:27:50.182441 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.182408 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv"] Jul 02 14:27:50.182890 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.182712 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d4ff04-0aaa-4989-9d49-725e5213cbf7" containerName="main" Jul 02 14:27:50.182890 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.182723 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d4ff04-0aaa-4989-9d49-725e5213cbf7" containerName="main" Jul 02 14:27:50.182890 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.182778 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="24d4ff04-0aaa-4989-9d49-725e5213cbf7" containerName="main" Jul 02 14:27:50.185583 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.185564 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.188262 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.188240 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\"" Jul 02 14:27:50.198863 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.198839 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv"] Jul 02 14:27:50.360995 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.360953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.360995 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.361005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj5k5\" (UniqueName: \"kubernetes.io/projected/e159f007-0efb-444b-9433-84cf81a85420-kube-api-access-kj5k5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.361221 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.361062 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.361221 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.361092 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.361221 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.361122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e159f007-0efb-444b-9433-84cf81a85420-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.361221 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.361145 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.462348 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.462264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.462348 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.462304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj5k5\" (UniqueName: \"kubernetes.io/projected/e159f007-0efb-444b-9433-84cf81a85420-kube-api-access-kj5k5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.462348 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.462335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.462619 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.462453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.462619 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.462483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e159f007-0efb-444b-9433-84cf81a85420-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.462619 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.462521 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.462756 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.462734 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.462867 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.462838 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.462908 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.462873 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.464646 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.464629 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.465309 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.465282 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e159f007-0efb-444b-9433-84cf81a85420-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.472312 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.472290 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj5k5\" (UniqueName: \"kubernetes.io/projected/e159f007-0efb-444b-9433-84cf81a85420-kube-api-access-kj5k5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.495231 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.495189 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:27:50.651741 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:50.651715 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv"] Jul 02 14:27:50.654013 ip-10-0-131-243 kubenswrapper[2578]: W0702 14:27:50.653985 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode159f007_0efb_444b_9433_84cf81a85420.slice/crio-64be3042e62075db929844f203fb9a16e7c1c4975317bab8cd51b5f85e07308a WatchSource:0}: Error finding container 64be3042e62075db929844f203fb9a16e7c1c4975317bab8cd51b5f85e07308a: Status 404 returned error can't find the container with id 64be3042e62075db929844f203fb9a16e7c1c4975317bab8cd51b5f85e07308a Jul 02 14:27:51.424900 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:51.424866 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" event={"ID":"e159f007-0efb-444b-9433-84cf81a85420","Type":"ContainerStarted","Data":"889e2406fab3ff40bf08c3e26d889b0c8d9047955103c5152ce501f24ce644ba"} Jul 02 14:27:51.425281 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:51.424937 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" event={"ID":"e159f007-0efb-444b-9433-84cf81a85420","Type":"ContainerStarted","Data":"64be3042e62075db929844f203fb9a16e7c1c4975317bab8cd51b5f85e07308a"} Jul 02 14:27:51.446100 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:51.446053 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" podStartSLOduration=1.4460379749999999 podStartE2EDuration="1.446037975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:27:51.443967757 +0000 UTC m=+4436.288681436" watchObservedRunningTime="2026-07-02 14:27:51.446037975 +0000 UTC m=+4436.290751630" Jul 02 14:27:51.496048 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:27:51.496026 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:28:00.495859 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:28:00.495817 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:28:00.508250 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:28:00.508221 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:28:01.484758 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:28:01.484730 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:28:55.411274 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:28:55.411239 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv"] Jul 02 14:28:55.909024 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:28:55.908856 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:28:55.920277 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:28:55.920257 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:29:10.431046 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.430980 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" podUID="e159f007-0efb-444b-9433-84cf81a85420" containerName="main" containerID="cri-o://889e2406fab3ff40bf08c3e26d889b0c8d9047955103c5152ce501f24ce644ba" gracePeriod=45 Jul 02 14:29:10.475205 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.475143 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" podUID="e159f007-0efb-444b-9433-84cf81a85420" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused" Jul 02 14:29:10.689654 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.689581 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:29:10.696493 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.696464 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-home\") pod \"e159f007-0efb-444b-9433-84cf81a85420\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " Jul 02 14:29:10.696601 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.696513 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e159f007-0efb-444b-9433-84cf81a85420-tls-certs\") pod \"e159f007-0efb-444b-9433-84cf81a85420\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " Jul 02 14:29:10.696601 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.696546 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj5k5\" (UniqueName: \"kubernetes.io/projected/e159f007-0efb-444b-9433-84cf81a85420-kube-api-access-kj5k5\") pod \"e159f007-0efb-444b-9433-84cf81a85420\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " Jul 02 14:29:10.696601 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.696569 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-tmp-dir\") pod \"e159f007-0efb-444b-9433-84cf81a85420\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " Jul 02 14:29:10.696721 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.696618 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-dshm\") pod \"e159f007-0efb-444b-9433-84cf81a85420\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " Jul 02 14:29:10.696721 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.696645 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-model-cache\") pod \"e159f007-0efb-444b-9433-84cf81a85420\" (UID: \"e159f007-0efb-444b-9433-84cf81a85420\") " Jul 02 14:29:10.696824 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.696756 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-home" (OuterVolumeSpecName: "home") pod "e159f007-0efb-444b-9433-84cf81a85420" (UID: "e159f007-0efb-444b-9433-84cf81a85420"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:29:10.696989 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.696949 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e159f007-0efb-444b-9433-84cf81a85420" (UID: "e159f007-0efb-444b-9433-84cf81a85420"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:29:10.696989 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.696981 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-model-cache" (OuterVolumeSpecName: "model-cache") pod "e159f007-0efb-444b-9433-84cf81a85420" (UID: "e159f007-0efb-444b-9433-84cf81a85420"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:29:10.697193 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.697172 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-tmp-dir\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:29:10.697252 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.697199 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-model-cache\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:29:10.697252 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.697213 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-home\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:29:10.699009 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.698985 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e159f007-0efb-444b-9433-84cf81a85420-kube-api-access-kj5k5" (OuterVolumeSpecName: "kube-api-access-kj5k5") pod "e159f007-0efb-444b-9433-84cf81a85420" (UID: "e159f007-0efb-444b-9433-84cf81a85420"). InnerVolumeSpecName "kube-api-access-kj5k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 14:29:10.699130 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.699008 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e159f007-0efb-444b-9433-84cf81a85420-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e159f007-0efb-444b-9433-84cf81a85420" (UID: "e159f007-0efb-444b-9433-84cf81a85420"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 14:29:10.699130 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.699089 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-dshm" (OuterVolumeSpecName: "dshm") pod "e159f007-0efb-444b-9433-84cf81a85420" (UID: "e159f007-0efb-444b-9433-84cf81a85420"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:29:10.779858 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.779818 2578 generic.go:356] "Generic (PLEG): container finished" podID="e159f007-0efb-444b-9433-84cf81a85420" containerID="889e2406fab3ff40bf08c3e26d889b0c8d9047955103c5152ce501f24ce644ba" exitCode=0 Jul 02 14:29:10.780042 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.779862 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" Jul 02 14:29:10.780042 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.779885 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" event={"ID":"e159f007-0efb-444b-9433-84cf81a85420","Type":"ContainerDied","Data":"889e2406fab3ff40bf08c3e26d889b0c8d9047955103c5152ce501f24ce644ba"} Jul 02 14:29:10.780042 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.779978 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv" event={"ID":"e159f007-0efb-444b-9433-84cf81a85420","Type":"ContainerDied","Data":"64be3042e62075db929844f203fb9a16e7c1c4975317bab8cd51b5f85e07308a"} Jul 02 14:29:10.780042 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.780009 2578 scope.go:117] "RemoveContainer" containerID="889e2406fab3ff40bf08c3e26d889b0c8d9047955103c5152ce501f24ce644ba" Jul 02 14:29:10.789638 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.789619 2578 scope.go:117] "RemoveContainer" containerID="889e2406fab3ff40bf08c3e26d889b0c8d9047955103c5152ce501f24ce644ba" Jul 02 14:29:10.789873 ip-10-0-131-243 kubenswrapper[2578]: E0702 14:29:10.789855 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889e2406fab3ff40bf08c3e26d889b0c8d9047955103c5152ce501f24ce644ba\": container with ID starting with 889e2406fab3ff40bf08c3e26d889b0c8d9047955103c5152ce501f24ce644ba not found: ID does not exist" containerID="889e2406fab3ff40bf08c3e26d889b0c8d9047955103c5152ce501f24ce644ba" Jul 02 14:29:10.789954 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.789879 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889e2406fab3ff40bf08c3e26d889b0c8d9047955103c5152ce501f24ce644ba"} err="failed to get container status \"889e2406fab3ff40bf08c3e26d889b0c8d9047955103c5152ce501f24ce644ba\": rpc error: code = NotFound desc = could not find container \"889e2406fab3ff40bf08c3e26d889b0c8d9047955103c5152ce501f24ce644ba\": container with ID starting with 889e2406fab3ff40bf08c3e26d889b0c8d9047955103c5152ce501f24ce644ba not found: ID does not exist" Jul 02 14:29:10.797675 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.797654 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e159f007-0efb-444b-9433-84cf81a85420-dshm\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:29:10.797742 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.797676 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e159f007-0efb-444b-9433-84cf81a85420-tls-certs\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:29:10.797742 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.797686 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj5k5\" (UniqueName: \"kubernetes.io/projected/e159f007-0efb-444b-9433-84cf81a85420-kube-api-access-kj5k5\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:29:10.802092 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.802068 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv"] Jul 02 14:29:10.808249 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:10.808226 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv"] Jul 02 14:29:11.625200 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:11.625157 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e159f007-0efb-444b-9433-84cf81a85420" path="/var/lib/kubelet/pods/e159f007-0efb-444b-9433-84cf81a85420/volumes" Jul 02 14:29:43.325661 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.325628 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47"] Jul 02 14:29:43.326203 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.326071 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e159f007-0efb-444b-9433-84cf81a85420" containerName="main" Jul 02 14:29:43.326203 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.326090 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e159f007-0efb-444b-9433-84cf81a85420" containerName="main" Jul 02 14:29:43.326203 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.326174 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e159f007-0efb-444b-9433-84cf81a85420" containerName="main" Jul 02 14:29:43.329271 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.329251 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.332438 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.332418 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\"" Jul 02 14:29:43.332577 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.332450 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-mpk8g\"" Jul 02 14:29:43.341992 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.341971 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47"] Jul 02 14:29:43.469093 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.469059 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.469252 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.469102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.469252 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.469136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.469252 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.469157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.469252 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.469174 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d19e82-da62-4a7d-bdc2-9453e7153c06-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.469390 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.469273 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb9tb\" (UniqueName: \"kubernetes.io/projected/18d19e82-da62-4a7d-bdc2-9453e7153c06-kube-api-access-lb9tb\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.569933 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.569884 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.570073 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.569964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.570073 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.569997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.570073 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.570025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.570073 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.570052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d19e82-da62-4a7d-bdc2-9453e7153c06-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.570295 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.570143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb9tb\" (UniqueName: \"kubernetes.io/projected/18d19e82-da62-4a7d-bdc2-9453e7153c06-kube-api-access-lb9tb\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.570338 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.570307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.570392 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.570365 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.570437 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.570389 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.572551 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.572525 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.572891 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.572869 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d19e82-da62-4a7d-bdc2-9453e7153c06-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.578811 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.578763 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb9tb\" (UniqueName: \"kubernetes.io/projected/18d19e82-da62-4a7d-bdc2-9453e7153c06-kube-api-access-lb9tb\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.639597 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.639576 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:43.780720 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.780693 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47"] Jul 02 14:29:43.782502 ip-10-0-131-243 kubenswrapper[2578]: W0702 14:29:43.782479 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d19e82_da62_4a7d_bdc2_9453e7153c06.slice/crio-eee3a97b4896e96c0607ea967972134a9fd6085be5adbf3529407f21736dd0b7 WatchSource:0}: Error finding container eee3a97b4896e96c0607ea967972134a9fd6085be5adbf3529407f21736dd0b7: Status 404 returned error can't find the container with id eee3a97b4896e96c0607ea967972134a9fd6085be5adbf3529407f21736dd0b7 Jul 02 14:29:43.784482 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.784465 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 14:29:43.925893 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:43.925812 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" event={"ID":"18d19e82-da62-4a7d-bdc2-9453e7153c06","Type":"ContainerStarted","Data":"eee3a97b4896e96c0607ea967972134a9fd6085be5adbf3529407f21736dd0b7"} Jul 02 14:29:44.932240 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:44.932205 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" event={"ID":"18d19e82-da62-4a7d-bdc2-9453e7153c06","Type":"ContainerStarted","Data":"ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd"} Jul 02 14:29:44.932651 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:44.932296 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:45.938469 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:45.938428 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" event={"ID":"18d19e82-da62-4a7d-bdc2-9453e7153c06","Type":"ContainerStarted","Data":"528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c"} Jul 02 14:29:45.959753 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:45.959698 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" podStartSLOduration=2.254422131 podStartE2EDuration="2.959680831s" totalImagesPullingTime="705.2587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:29:45.959180607 +0000 UTC m=+4550.803894286" watchObservedRunningTime="2026-07-02 14:29:45.959680831 +0000 UTC m=+4550.804394512" Jul 02 14:29:46.640423 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:46.640388 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:53.639979 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:53.639944 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:53.652409 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:53.652386 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:53.989954 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:53.989853 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:29:54.996429 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:29:54.996404 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:31:04.837462 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:04.837426 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47"] Jul 02 14:31:19.860697 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:19.860655 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" podUID="18d19e82-da62-4a7d-bdc2-9453e7153c06" containerName="main" containerID="cri-o://528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c" gracePeriod=45 Jul 02 14:31:19.972184 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:19.972149 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" podUID="18d19e82-da62-4a7d-bdc2-9453e7153c06" containerName="llm-d-routing-sidecar" containerID="cri-o://ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd" gracePeriod=45 Jul 02 14:31:19.979829 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:19.979796 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" podUID="18d19e82-da62-4a7d-bdc2-9453e7153c06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused" Jul 02 14:31:20.222151 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.222125 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:31:20.231323 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.231305 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-home\") pod \"18d19e82-da62-4a7d-bdc2-9453e7153c06\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " Jul 02 14:31:20.231398 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.231352 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d19e82-da62-4a7d-bdc2-9453e7153c06-tls-certs\") pod \"18d19e82-da62-4a7d-bdc2-9453e7153c06\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " Jul 02 14:31:20.231398 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.231372 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-dshm\") pod \"18d19e82-da62-4a7d-bdc2-9453e7153c06\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " Jul 02 14:31:20.231398 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.231391 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-tmp-dir\") pod \"18d19e82-da62-4a7d-bdc2-9453e7153c06\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " Jul 02 14:31:20.231511 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.231422 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-model-cache\") pod \"18d19e82-da62-4a7d-bdc2-9453e7153c06\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " Jul 02 14:31:20.231566 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.231503 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-home" (OuterVolumeSpecName: "home") pod "18d19e82-da62-4a7d-bdc2-9453e7153c06" (UID: "18d19e82-da62-4a7d-bdc2-9453e7153c06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:31:20.231625 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.231575 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb9tb\" (UniqueName: \"kubernetes.io/projected/18d19e82-da62-4a7d-bdc2-9453e7153c06-kube-api-access-lb9tb\") pod \"18d19e82-da62-4a7d-bdc2-9453e7153c06\" (UID: \"18d19e82-da62-4a7d-bdc2-9453e7153c06\") " Jul 02 14:31:20.231677 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.231646 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18d19e82-da62-4a7d-bdc2-9453e7153c06" (UID: "18d19e82-da62-4a7d-bdc2-9453e7153c06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:31:20.231677 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.231664 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-model-cache" (OuterVolumeSpecName: "model-cache") pod "18d19e82-da62-4a7d-bdc2-9453e7153c06" (UID: "18d19e82-da62-4a7d-bdc2-9453e7153c06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:31:20.231851 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.231835 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-home\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:31:20.231910 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.231858 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-tmp-dir\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:31:20.231910 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.231872 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-model-cache\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:31:20.233457 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.233437 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d19e82-da62-4a7d-bdc2-9453e7153c06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18d19e82-da62-4a7d-bdc2-9453e7153c06" (UID: "18d19e82-da62-4a7d-bdc2-9453e7153c06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 14:31:20.233457 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.233448 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-dshm" (OuterVolumeSpecName: "dshm") pod "18d19e82-da62-4a7d-bdc2-9453e7153c06" (UID: "18d19e82-da62-4a7d-bdc2-9453e7153c06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:31:20.233577 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.233559 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d19e82-da62-4a7d-bdc2-9453e7153c06-kube-api-access-lb9tb" (OuterVolumeSpecName: "kube-api-access-lb9tb") pod "18d19e82-da62-4a7d-bdc2-9453e7153c06" (UID: "18d19e82-da62-4a7d-bdc2-9453e7153c06"). InnerVolumeSpecName "kube-api-access-lb9tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 14:31:20.332539 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.332499 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d19e82-da62-4a7d-bdc2-9453e7153c06-tls-certs\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:31:20.332539 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.332531 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d19e82-da62-4a7d-bdc2-9453e7153c06-dshm\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:31:20.332539 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.332542 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb9tb\" (UniqueName: \"kubernetes.io/projected/18d19e82-da62-4a7d-bdc2-9453e7153c06-kube-api-access-lb9tb\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:31:20.362292 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.362256 2578 generic.go:356] "Generic (PLEG): container finished" podID="18d19e82-da62-4a7d-bdc2-9453e7153c06" containerID="528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c" exitCode=0 Jul 02 14:31:20.362292 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.362284 2578 generic.go:356] "Generic (PLEG): container finished" podID="18d19e82-da62-4a7d-bdc2-9453e7153c06" containerID="ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd" exitCode=0 Jul 02 14:31:20.362522 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.362307 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" Jul 02 14:31:20.362522 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.362313 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" event={"ID":"18d19e82-da62-4a7d-bdc2-9453e7153c06","Type":"ContainerDied","Data":"528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c"} Jul 02 14:31:20.362522 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.362370 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" event={"ID":"18d19e82-da62-4a7d-bdc2-9453e7153c06","Type":"ContainerDied","Data":"ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd"} Jul 02 14:31:20.362522 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.362394 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47" event={"ID":"18d19e82-da62-4a7d-bdc2-9453e7153c06","Type":"ContainerDied","Data":"eee3a97b4896e96c0607ea967972134a9fd6085be5adbf3529407f21736dd0b7"} Jul 02 14:31:20.362522 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.362421 2578 scope.go:117] "RemoveContainer" containerID="528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c" Jul 02 14:31:20.371368 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.371349 2578 scope.go:117] "RemoveContainer" containerID="ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd" Jul 02 14:31:20.378934 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.378900 2578 scope.go:117] "RemoveContainer" containerID="528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c" Jul 02 14:31:20.379217 ip-10-0-131-243 kubenswrapper[2578]: E0702 14:31:20.379196 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c\": container with ID starting with 528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c not found: ID does not exist" containerID="528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c" Jul 02 14:31:20.379272 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.379225 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c"} err="failed to get container status \"528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c\": rpc error: code = NotFound desc = could not find container \"528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c\": container with ID starting with 528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c not found: ID does not exist" Jul 02 14:31:20.379272 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.379242 2578 scope.go:117] "RemoveContainer" containerID="ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd" Jul 02 14:31:20.379477 ip-10-0-131-243 kubenswrapper[2578]: E0702 14:31:20.379460 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd\": container with ID starting with ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd not found: ID does not exist" containerID="ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd" Jul 02 14:31:20.379518 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.379482 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd"} err="failed to get container status \"ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd\": rpc error: code = NotFound desc = could not find container \"ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd\": container with ID starting with ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd not found: ID does not exist" Jul 02 14:31:20.379518 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.379497 2578 scope.go:117] "RemoveContainer" containerID="528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c" Jul 02 14:31:20.379739 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.379718 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c"} err="failed to get container status \"528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c\": rpc error: code = NotFound desc = could not find container \"528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c\": container with ID starting with 528a76661e890f0e67c0336af3e038fe69a480ddb8e59ffa6dbc54368fe30e9c not found: ID does not exist" Jul 02 14:31:20.379778 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.379741 2578 scope.go:117] "RemoveContainer" containerID="ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd" Jul 02 14:31:20.379981 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.379961 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd"} err="failed to get container status \"ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd\": rpc error: code = NotFound desc = could not find container \"ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd\": container with ID starting with ce9bdf0fbc6beb6984f8f0cd9495abc795f33e5babbc879eaa9d2b2c88a3e9cd not found: ID does not exist" Jul 02 14:31:20.386468 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.386443 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47"] Jul 02 14:31:20.392239 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:20.392185 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47"] Jul 02 14:31:21.625269 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:31:21.625233 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18d19e82-da62-4a7d-bdc2-9453e7153c06" path="/var/lib/kubelet/pods/18d19e82-da62-4a7d-bdc2-9453e7153c06/volumes" Jul 02 14:33:26.981708 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:26.981673 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7"] Jul 02 14:33:26.982137 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:26.982028 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18d19e82-da62-4a7d-bdc2-9453e7153c06" containerName="main" Jul 02 14:33:26.982137 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:26.982041 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="18d19e82-da62-4a7d-bdc2-9453e7153c06" containerName="main" Jul 02 14:33:26.982137 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:26.982060 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18d19e82-da62-4a7d-bdc2-9453e7153c06" containerName="llm-d-routing-sidecar" Jul 02 14:33:26.982137 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:26.982066 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="18d19e82-da62-4a7d-bdc2-9453e7153c06" containerName="llm-d-routing-sidecar" Jul 02 14:33:26.982137 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:26.982130 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="18d19e82-da62-4a7d-bdc2-9453e7153c06" containerName="llm-d-routing-sidecar" Jul 02 14:33:26.982137 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:26.982138 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="18d19e82-da62-4a7d-bdc2-9453e7153c06" containerName="main" Jul 02 14:33:26.985090 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:26.985073 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:26.987660 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:26.987639 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\"" Jul 02 14:33:26.995059 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:26.995036 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7"] Jul 02 14:33:27.056657 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.056632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.056765 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.056673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ffeb8f-af04-4c04-8729-562b433d5689-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.056765 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.056697 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.056765 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.056736 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcrld\" (UniqueName: \"kubernetes.io/projected/31ffeb8f-af04-4c04-8729-562b433d5689-kube-api-access-vcrld\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.056871 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.056815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.056871 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.056831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.157974 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.157944 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ffeb8f-af04-4c04-8729-562b433d5689-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.158086 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.157984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.158086 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.158000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcrld\" (UniqueName: \"kubernetes.io/projected/31ffeb8f-af04-4c04-8729-562b433d5689-kube-api-access-vcrld\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.158086 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.158044 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.158086 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.158060 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.158286 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.158098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.158406 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.158383 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.158465 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.158415 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.158623 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.158603 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.160403 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.160371 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.160509 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.160378 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ffeb8f-af04-4c04-8729-562b433d5689-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.168908 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.168886 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcrld\" (UniqueName: \"kubernetes.io/projected/31ffeb8f-af04-4c04-8729-562b433d5689-kube-api-access-vcrld\") pod \"rolling-upgrade-test-kserve-859df57bcb-625z7\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.295673 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.295608 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:27.424968 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.424942 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7"] Jul 02 14:33:27.426998 ip-10-0-131-243 kubenswrapper[2578]: W0702 14:33:27.426971 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ffeb8f_af04_4c04_8729_562b433d5689.slice/crio-3ab1004f5e9d662afa969bb1e78d4b2329956df86322f902e339585e56bb7f00 WatchSource:0}: Error finding container 3ab1004f5e9d662afa969bb1e78d4b2329956df86322f902e339585e56bb7f00: Status 404 returned error can't find the container with id 3ab1004f5e9d662afa969bb1e78d4b2329956df86322f902e339585e56bb7f00 Jul 02 14:33:27.923575 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.923543 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" event={"ID":"31ffeb8f-af04-4c04-8729-562b433d5689","Type":"ContainerStarted","Data":"aaa07a3f92ed205cce49cd415f5b2bb14eebccbe71a637bd5e58ac38ebc7a2c5"} Jul 02 14:33:27.923748 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.923590 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" event={"ID":"31ffeb8f-af04-4c04-8729-562b433d5689","Type":"ContainerStarted","Data":"3ab1004f5e9d662afa969bb1e78d4b2329956df86322f902e339585e56bb7f00"} Jul 02 14:33:27.942395 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:27.942348 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" podStartSLOduration=1.9423330559999998 podStartE2EDuration="1.942333056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:33:27.941711027 +0000 UTC m=+4772.786424703" watchObservedRunningTime="2026-07-02 14:33:27.942333056 +0000 UTC m=+4772.787046731" Jul 02 14:33:28.295744 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:28.295669 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:37.295864 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:37.295826 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:37.308265 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:37.308243 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:37.976733 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:37.976703 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:33:51.347870 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:51.347837 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7"] Jul 02 14:33:55.935013 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:55.934898 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:33:55.946616 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:33:55.946594 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:34:06.369720 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.369661 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" podUID="31ffeb8f-af04-4c04-8729-562b433d5689" containerName="main" containerID="cri-o://aaa07a3f92ed205cce49cd415f5b2bb14eebccbe71a637bd5e58ac38ebc7a2c5" gracePeriod=45 Jul 02 14:34:06.614312 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.614289 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:34:06.669225 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.669152 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ffeb8f-af04-4c04-8729-562b433d5689-tls-certs\") pod \"31ffeb8f-af04-4c04-8729-562b433d5689\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " Jul 02 14:34:06.669225 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.669193 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcrld\" (UniqueName: \"kubernetes.io/projected/31ffeb8f-af04-4c04-8729-562b433d5689-kube-api-access-vcrld\") pod \"31ffeb8f-af04-4c04-8729-562b433d5689\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " Jul 02 14:34:06.669225 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.669215 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-model-cache\") pod \"31ffeb8f-af04-4c04-8729-562b433d5689\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " Jul 02 14:34:06.669486 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.669230 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-dshm\") pod \"31ffeb8f-af04-4c04-8729-562b433d5689\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " Jul 02 14:34:06.669486 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.669315 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-home\") pod \"31ffeb8f-af04-4c04-8729-562b433d5689\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " Jul 02 14:34:06.669486 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.669353 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-tmp-dir\") pod \"31ffeb8f-af04-4c04-8729-562b433d5689\" (UID: \"31ffeb8f-af04-4c04-8729-562b433d5689\") " Jul 02 14:34:06.669641 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.669506 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-model-cache" (OuterVolumeSpecName: "model-cache") pod "31ffeb8f-af04-4c04-8729-562b433d5689" (UID: "31ffeb8f-af04-4c04-8729-562b433d5689"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:34:06.669641 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.669594 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-home" (OuterVolumeSpecName: "home") pod "31ffeb8f-af04-4c04-8729-562b433d5689" (UID: "31ffeb8f-af04-4c04-8729-562b433d5689"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:34:06.669745 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.669639 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31ffeb8f-af04-4c04-8729-562b433d5689" (UID: "31ffeb8f-af04-4c04-8729-562b433d5689"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:34:06.669745 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.669716 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-model-cache\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:34:06.669745 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.669732 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-home\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:34:06.669745 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.669741 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-tmp-dir\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:34:06.671285 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.671264 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ffeb8f-af04-4c04-8729-562b433d5689-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31ffeb8f-af04-4c04-8729-562b433d5689" (UID: "31ffeb8f-af04-4c04-8729-562b433d5689"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 14:34:06.671719 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.671694 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-dshm" (OuterVolumeSpecName: "dshm") pod "31ffeb8f-af04-4c04-8729-562b433d5689" (UID: "31ffeb8f-af04-4c04-8729-562b433d5689"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 14:34:06.671719 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.671710 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ffeb8f-af04-4c04-8729-562b433d5689-kube-api-access-vcrld" (OuterVolumeSpecName: "kube-api-access-vcrld") pod "31ffeb8f-af04-4c04-8729-562b433d5689" (UID: "31ffeb8f-af04-4c04-8729-562b433d5689"). InnerVolumeSpecName "kube-api-access-vcrld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 14:34:06.771116 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.771085 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ffeb8f-af04-4c04-8729-562b433d5689-tls-certs\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:34:06.771116 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.771113 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcrld\" (UniqueName: \"kubernetes.io/projected/31ffeb8f-af04-4c04-8729-562b433d5689-kube-api-access-vcrld\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:34:06.771296 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:06.771125 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ffeb8f-af04-4c04-8729-562b433d5689-dshm\") on node \"ip-10-0-131-243.ec2.internal\" DevicePath \"\"" Jul 02 14:34:07.099887 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:07.099851 2578 generic.go:356] "Generic (PLEG): container finished" podID="31ffeb8f-af04-4c04-8729-562b433d5689" containerID="aaa07a3f92ed205cce49cd415f5b2bb14eebccbe71a637bd5e58ac38ebc7a2c5" exitCode=0 Jul 02 14:34:07.100090 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:07.099901 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" Jul 02 14:34:07.100090 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:07.099945 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" event={"ID":"31ffeb8f-af04-4c04-8729-562b433d5689","Type":"ContainerDied","Data":"aaa07a3f92ed205cce49cd415f5b2bb14eebccbe71a637bd5e58ac38ebc7a2c5"} Jul 02 14:34:07.100090 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:07.099997 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7" event={"ID":"31ffeb8f-af04-4c04-8729-562b433d5689","Type":"ContainerDied","Data":"3ab1004f5e9d662afa969bb1e78d4b2329956df86322f902e339585e56bb7f00"} Jul 02 14:34:07.100090 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:07.100020 2578 scope.go:117] "RemoveContainer" containerID="aaa07a3f92ed205cce49cd415f5b2bb14eebccbe71a637bd5e58ac38ebc7a2c5" Jul 02 14:34:07.109443 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:07.109422 2578 scope.go:117] "RemoveContainer" containerID="aaa07a3f92ed205cce49cd415f5b2bb14eebccbe71a637bd5e58ac38ebc7a2c5" Jul 02 14:34:07.109704 ip-10-0-131-243 kubenswrapper[2578]: E0702 14:34:07.109683 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa07a3f92ed205cce49cd415f5b2bb14eebccbe71a637bd5e58ac38ebc7a2c5\": container with ID starting with aaa07a3f92ed205cce49cd415f5b2bb14eebccbe71a637bd5e58ac38ebc7a2c5 not found: ID does not exist" containerID="aaa07a3f92ed205cce49cd415f5b2bb14eebccbe71a637bd5e58ac38ebc7a2c5" Jul 02 14:34:07.109797 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:07.109709 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa07a3f92ed205cce49cd415f5b2bb14eebccbe71a637bd5e58ac38ebc7a2c5"} err="failed to get container status \"aaa07a3f92ed205cce49cd415f5b2bb14eebccbe71a637bd5e58ac38ebc7a2c5\": rpc error: code = NotFound desc = could not find container \"aaa07a3f92ed205cce49cd415f5b2bb14eebccbe71a637bd5e58ac38ebc7a2c5\": container with ID starting with aaa07a3f92ed205cce49cd415f5b2bb14eebccbe71a637bd5e58ac38ebc7a2c5 not found: ID does not exist" Jul 02 14:34:07.121933 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:07.121898 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7"] Jul 02 14:34:07.125442 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:07.125421 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7"] Jul 02 14:34:07.625305 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:34:07.625265 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ffeb8f-af04-4c04-8729-562b433d5689" path="/var/lib/kubelet/pods/31ffeb8f-af04-4c04-8729-562b433d5689/volumes" Jul 02 14:37:11.445079 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:11.445047 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:13.096715 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:13.096685 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:14.735467 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:14.735438 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:16.364012 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:16.363979 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:18.022683 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:18.022643 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:19.597413 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:19.597372 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:21.299840 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:21.299805 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:23.262784 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:23.262749 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:24.889234 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:24.889202 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:26.551308 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:26.551270 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:28.131123 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:28.131091 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:29.739219 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:29.739184 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:31.346388 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:31.346353 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:32.980460 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:32.980429 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:34.581582 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:34.581545 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:36.201014 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:36.200976 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd_3dd126ed-b241-4f3f-bc01-3e3abe1b7a01/main/0.log" Jul 02 14:37:38.281258 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:38.281225 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbgnz_f2e2e7b8-5c47-41c9-8950-6bb74bb14be4/istio-proxy/0.log" Jul 02 14:37:38.321063 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:38.321033 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbgnz_f2e2e7b8-5c47-41c9-8950-6bb74bb14be4/istio-proxy/1.log" Jul 02 14:37:39.870090 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:39.870056 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbgnz_f2e2e7b8-5c47-41c9-8950-6bb74bb14be4/istio-proxy/0.log" Jul 02 14:37:39.914460 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:39.914431 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbgnz_f2e2e7b8-5c47-41c9-8950-6bb74bb14be4/istio-proxy/1.log" Jul 02 14:37:41.480811 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:41.480782 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbgnz_f2e2e7b8-5c47-41c9-8950-6bb74bb14be4/istio-proxy/0.log" Jul 02 14:37:41.519479 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:41.519445 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbgnz_f2e2e7b8-5c47-41c9-8950-6bb74bb14be4/istio-proxy/1.log" Jul 02 14:37:43.052272 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:43.052227 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbgnz_f2e2e7b8-5c47-41c9-8950-6bb74bb14be4/istio-proxy/0.log" Jul 02 14:37:43.092939 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:43.092897 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbgnz_f2e2e7b8-5c47-41c9-8950-6bb74bb14be4/istio-proxy/1.log" Jul 02 14:37:44.182424 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:44.182393 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:37:44.400817 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:44.400781 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:37:45.539439 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:45.539406 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:37:45.762510 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:45.762480 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:37:46.932261 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:46.932230 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:37:47.123360 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:47.123286 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:37:48.335302 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:48.335275 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:37:48.570609 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:48.570578 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:37:49.694488 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:49.694461 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:37:49.919122 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:49.919090 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:37:51.059261 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:51.059233 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:37:51.283367 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:51.283337 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:37:52.431321 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:52.431290 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:37:52.658855 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:52.658827 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:37:53.795286 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:53.795259 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:37:54.021571 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:54.021534 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:37:55.144807 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:55.144777 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:37:55.376755 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:55.376727 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:37:56.553820 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:56.553791 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:37:56.779688 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:56.779657 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:37:57.957663 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:57.957633 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:37:58.179063 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:58.179037 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:37:59.310377 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:59.310347 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:37:59.524670 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:37:59.524644 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:38:00.658660 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:00.658630 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:38:00.875973 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:00.875942 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:38:02.026836 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:02.026808 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:38:02.246453 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:02.246428 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:38:03.379777 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:03.379750 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:38:03.604716 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:03.604688 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:38:04.709316 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:04.709284 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:38:04.931241 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:04.931208 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:38:06.061780 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:06.061753 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:38:06.283965 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:06.283933 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:38:07.485781 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:07.485753 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:38:07.713300 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:07.713266 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:38:08.891997 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:08.891965 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:38:09.129927 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:09.129898 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:38:10.270256 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:10.270226 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:38:10.459441 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:10.459405 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:38:11.715584 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:11.715557 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:38:11.935512 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:11.935480 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:38:17.732955 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:17.732910 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j8m57_c443c208-241a-4759-a5d9-657a8ff485c2/global-pull-secret-syncer/0.log" Jul 02 14:38:17.886199 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:17.886172 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rqfvz_9a98330d-24e4-490f-810a-bc2488b8e20e/konnectivity-agent/0.log" Jul 02 14:38:17.964675 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:17.964648 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-243.ec2.internal_41c092df77332bb9c2b6523d6044f2bf/haproxy/0.log" Jul 02 14:38:24.010861 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:24.010830 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-bn8mr_73fb5ed2-6f62-452c-8f2e-9a1d40a4411d/kuadrant-console-plugin/0.log" Jul 02 14:38:24.255543 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:24.255516 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-6f84d_3cf77edd-bbc3-4292-a811-54328c9d837d/limitador/0.log" Jul 02 14:38:25.681628 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:25.681541 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-54cb84584-nd45d_6ec53108-0c5d-4c8b-a563-74ab03c784f5/cluster-monitoring-operator/0.log" Jul 02 14:38:25.934029 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:25.933941 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2jxh_41a0736c-8423-45ec-a26b-e8ba78af9526/node-exporter/0.log" Jul 02 14:38:25.958653 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:25.958631 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2jxh_41a0736c-8423-45ec-a26b-e8ba78af9526/kube-rbac-proxy/0.log" Jul 02 14:38:25.983817 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:25.983796 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2jxh_41a0736c-8423-45ec-a26b-e8ba78af9526/init-textfile/0.log" Jul 02 14:38:28.127014 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.126978 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq"] Jul 02 14:38:28.127388 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.127273 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ffeb8f-af04-4c04-8729-562b433d5689" containerName="main" Jul 02 14:38:28.127388 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.127284 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ffeb8f-af04-4c04-8729-562b433d5689" containerName="main" Jul 02 14:38:28.127388 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.127349 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ffeb8f-af04-4c04-8729-562b433d5689" containerName="main" Jul 02 14:38:28.130281 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.130259 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.133103 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.133080 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfr7l\"/\"openshift-service-ca.crt\"" Jul 02 14:38:28.133219 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.133080 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfr7l\"/\"kube-root-ca.crt\"" Jul 02 14:38:28.133219 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.133158 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zfr7l\"/\"default-dockercfg-cvgg4\"" Jul 02 14:38:28.142281 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.142258 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq"] Jul 02 14:38:28.256684 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.256658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df913ab7-b762-400c-a238-672edbc4a5fc-proc\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.256811 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.256706 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcfcm\" (UniqueName: \"kubernetes.io/projected/df913ab7-b762-400c-a238-672edbc4a5fc-kube-api-access-gcfcm\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.256811 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.256730 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df913ab7-b762-400c-a238-672edbc4a5fc-sys\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.256908 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.256802 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df913ab7-b762-400c-a238-672edbc4a5fc-lib-modules\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.256908 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.256841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df913ab7-b762-400c-a238-672edbc4a5fc-podres\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.272872 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.272850 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/2.log" Jul 02 14:38:28.280708 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.280680 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-c7h5s_1c82d88b-a9b9-4af2-b988-c6374ee4a034/console-operator/3.log" Jul 02 14:38:28.358159 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.358136 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df913ab7-b762-400c-a238-672edbc4a5fc-proc\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.358272 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.358198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfcm\" (UniqueName: \"kubernetes.io/projected/df913ab7-b762-400c-a238-672edbc4a5fc-kube-api-access-gcfcm\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.358272 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.358223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df913ab7-b762-400c-a238-672edbc4a5fc-sys\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.358272 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.358244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df913ab7-b762-400c-a238-672edbc4a5fc-lib-modules\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.358272 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.358265 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df913ab7-b762-400c-a238-672edbc4a5fc-proc\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.358485 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.358273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df913ab7-b762-400c-a238-672edbc4a5fc-podres\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.358485 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.358326 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df913ab7-b762-400c-a238-672edbc4a5fc-sys\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.358485 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.358376 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df913ab7-b762-400c-a238-672edbc4a5fc-podres\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.358485 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.358385 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df913ab7-b762-400c-a238-672edbc4a5fc-lib-modules\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.367014 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.366995 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcfcm\" (UniqueName: \"kubernetes.io/projected/df913ab7-b762-400c-a238-672edbc4a5fc-kube-api-access-gcfcm\") pod \"perf-node-gather-daemonset-hpjqq\" (UID: \"df913ab7-b762-400c-a238-672edbc4a5fc\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.440170 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.440123 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:28.569748 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:28.569663 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq"] Jul 02 14:38:28.572457 ip-10-0-131-243 kubenswrapper[2578]: W0702 14:38:28.572423 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddf913ab7_b762_400c_a238_672edbc4a5fc.slice/crio-b1744fe68148b9b3e17a5f6eb2bf102eca02d41f2307240da64aa259f9ddf17c WatchSource:0}: Error finding container b1744fe68148b9b3e17a5f6eb2bf102eca02d41f2307240da64aa259f9ddf17c: Status 404 returned error can't find the container with id b1744fe68148b9b3e17a5f6eb2bf102eca02d41f2307240da64aa259f9ddf17c Jul 02 14:38:29.268082 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:29.268043 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" event={"ID":"df913ab7-b762-400c-a238-672edbc4a5fc","Type":"ContainerStarted","Data":"28029c6bcd070c91c0c40e8c228ab3ec4ed5a3717a8936fdf73e0113ca93d9b6"} Jul 02 14:38:29.268468 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:29.268102 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" event={"ID":"df913ab7-b762-400c-a238-672edbc4a5fc","Type":"ContainerStarted","Data":"b1744fe68148b9b3e17a5f6eb2bf102eca02d41f2307240da64aa259f9ddf17c"} Jul 02 14:38:29.268468 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:29.268132 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:29.287058 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:29.287010 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" podStartSLOduration=1.286997392 podStartE2EDuration="1.286997392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:38:29.286248564 +0000 UTC m=+5074.130962264" watchObservedRunningTime="2026-07-02 14:38:29.286997392 +0000 UTC m=+5074.131711070" Jul 02 14:38:29.466624 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:29.466597 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-69d9f679db-fr8d5_5006a319-17cb-4694-9d11-94f89ad9dfe9/volume-data-source-validator/0.log" Jul 02 14:38:30.573113 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:30.573080 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8825z_fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6/dns/0.log" Jul 02 14:38:30.598535 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:30.598506 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8825z_fa757129-f6f3-4ce7-8bb1-79dfe3cc47a6/kube-rbac-proxy/0.log" Jul 02 14:38:30.770493 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:30.770470 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rf9bz_f9481a84-1265-415b-8a3e-d2c8bfff0ada/dns-node-resolver/0.log" Jul 02 14:38:31.307932 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:31.307889 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8f89cb6f4-mhk48_4b1379fe-4206-4604-a25a-b036a50f866f/registry/0.log" Jul 02 14:38:31.365082 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:31.365048 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c4k2p_305c0bc7-b833-4bca-8f4e-6f4a392f09b0/node-ca/0.log" Jul 02 14:38:32.945708 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:32.945674 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbgnz_f2e2e7b8-5c47-41c9-8950-6bb74bb14be4/istio-proxy/0.log" Jul 02 14:38:32.984308 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:32.984279 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbgnz_f2e2e7b8-5c47-41c9-8950-6bb74bb14be4/istio-proxy/1.log" Jul 02 14:38:33.520038 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:33.520005 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-87lm7_c5f79ccc-8051-4f78-b824-b1c8c6652212/serve-healthcheck-canary/0.log" Jul 02 14:38:34.046470 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:34.046436 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-5qxxr_e38ece80-57ff-4bbf-a9fe-15721b60478c/insights-operator/0.log" Jul 02 14:38:34.049143 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:34.049115 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-5qxxr_e38ece80-57ff-4bbf-a9fe-15721b60478c/insights-operator/1.log" Jul 02 14:38:34.071341 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:34.071318 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l5xd6_07a2217b-3ced-42fd-9849-f3300d67d8b9/kube-rbac-proxy/0.log" Jul 02 14:38:34.094293 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:34.094270 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l5xd6_07a2217b-3ced-42fd-9849-f3300d67d8b9/exporter/0.log" Jul 02 14:38:34.117307 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:34.117289 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l5xd6_07a2217b-3ced-42fd-9849-f3300d67d8b9/extractor/0.log" Jul 02 14:38:35.282519 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:35.282492 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-hpjqq" Jul 02 14:38:37.142123 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:37.142031 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-676677444b-jz7rv_cca6b6ab-2b30-432c-8e4f-f9fad835f301/manager/0.log" Jul 02 14:38:37.196132 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:37.196105 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-q9cvl_0082f696-05fe-49e4-b13c-ed5b85b1958d/openshift-lws-operator/0.log" Jul 02 14:38:39.091901 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:39.091868 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-5zxnj_415e88c8-d5c2-445a-9ebe-8befb5520084/seaweedfs/0.log" Jul 02 14:38:40.027992 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:40.027950 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbgnz_f2e2e7b8-5c47-41c9-8950-6bb74bb14be4/istio-proxy/0.log" Jul 02 14:38:40.068999 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:40.068964 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbgnz_f2e2e7b8-5c47-41c9-8950-6bb74bb14be4/istio-proxy/1.log" Jul 02 14:38:41.704458 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:41.704387 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbgnz_f2e2e7b8-5c47-41c9-8950-6bb74bb14be4/istio-proxy/0.log" Jul 02 14:38:41.748225 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:41.748173 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbgnz_f2e2e7b8-5c47-41c9-8950-6bb74bb14be4/istio-proxy/1.log" Jul 02 14:38:45.421464 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:45.421428 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4p9hq_493bc598-29fb-4757-b9a7-b8bb1b346b77/kube-multus/0.log" Jul 02 14:38:45.787088 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:45.787006 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvkd2_d2787138-2777-487c-89da-f039c1f41ad2/kube-multus-additional-cni-plugins/0.log" Jul 02 14:38:45.809384 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:45.809357 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvkd2_d2787138-2777-487c-89da-f039c1f41ad2/egress-router-binary-copy/0.log" Jul 02 14:38:45.835460 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:45.835432 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvkd2_d2787138-2777-487c-89da-f039c1f41ad2/cni-plugins/0.log" Jul 02 14:38:45.857223 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:45.857203 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvkd2_d2787138-2777-487c-89da-f039c1f41ad2/bond-cni-plugin/0.log" Jul 02 14:38:45.882735 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:45.882715 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvkd2_d2787138-2777-487c-89da-f039c1f41ad2/routeoverride-cni/0.log" Jul 02 14:38:45.906066 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:45.906044 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvkd2_d2787138-2777-487c-89da-f039c1f41ad2/whereabouts-cni-bincopy/0.log" Jul 02 14:38:45.930407 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:45.930383 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvkd2_d2787138-2777-487c-89da-f039c1f41ad2/whereabouts-cni/0.log" Jul 02 14:38:45.960696 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:45.960663 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nmc96_ac5d1704-1da6-48e2-b5fa-edecec3a5ee4/network-metrics-daemon/0.log" Jul 02 14:38:45.979703 ip-10-0-131-243 kubenswrapper[2578]: I0702 14:38:45.979681 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nmc96_ac5d1704-1da6-48e2-b5fa-edecec3a5ee4/kube-rbac-proxy/0.log"