Jul 16 11:01:37.948198 ip-10-0-142-15 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 11:01:37.948209 ip-10-0-142-15 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 11:01:37.948216 ip-10-0-142-15 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 11:01:37.948439 ip-10-0-142-15 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 11:01:47.988398 ip-10-0-142-15 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 16 11:01:47.988416 ip-10-0-142-15 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b0467b7098f64f0ebd9e736e7f986298 -- Jul 16 11:03:48.051861 ip-10-0-142-15 systemd[1]: Starting Kubernetes Kubelet... Jul 16 11:03:48.487920 ip-10-0-142-15 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 11:03:48.487920 ip-10-0-142-15 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 11:03:48.487920 ip-10-0-142-15 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 11:03:48.487920 ip-10-0-142-15 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 11:03:48.487920 ip-10-0-142-15 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 11:03:48.487920 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.464678 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 11:03:48.487920 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.470992 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 11:03:48.487920 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471009 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 11:03:48.487920 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471013 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 11:03:48.487920 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471016 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 11:03:48.487920 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471019 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 11:03:48.487920 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471022 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 11:03:48.487920 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471024 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 11:03:48.487920 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471027 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471030 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471032 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471035 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471038 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471041 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471043 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471046 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471049 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471051 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471055 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471060 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471063 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471065 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471068 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471071 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471073 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471076 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471079 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471082 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 11:03:48.488765 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471085 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471088 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471090 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471093 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471096 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471100 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471102 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471105 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471108 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471110 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471113 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471116 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471119 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471121 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471124 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471127 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471130 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471132 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471135 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471137 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 11:03:48.489389 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471139 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471142 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471144 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471147 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471150 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471153 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471156 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471158 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471160 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471163 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471165 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471169 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471171 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471174 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471176 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471179 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471181 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471183 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471186 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471188 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 11:03:48.490034 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471191 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471193 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471196 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471198 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471201 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471203 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471206 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471208 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471210 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471213 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471217 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471220 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471223 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471226 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471228 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471232 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471236 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471239 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471243 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 11:03:48.490680 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471246 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471249 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471251 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471254 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471257 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471260 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471263 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471265 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471268 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471271 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471274 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471278 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.471282 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472002 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472009 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472013 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472016 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472019 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472022 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472026 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 11:03:48.491310 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472028 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472031 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472034 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472036 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472039 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472042 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472044 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472047 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472049 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472052 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472055 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472058 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472061 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472063 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472066 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472070 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472075 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472078 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472081 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472083 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 11:03:48.491973 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472086 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472088 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472091 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472094 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472097 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472100 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472102 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472105 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472107 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472110 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472113 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472115 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472118 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472120 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472123 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472126 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472130 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472133 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472137 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 11:03:48.492854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472140 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472143 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472146 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472148 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472151 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472154 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472157 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472159 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472161 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472165 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472167 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472170 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472173 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472175 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472178 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472180 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472182 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472185 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472188 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 11:03:48.493508 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472190 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472193 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472198 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472203 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472206 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472209 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472212 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472215 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472218 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472220 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472223 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472225 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472229 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472233 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472236 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472238 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472241 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472243 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472246 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 11:03:48.494160 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472249 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472252 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472254 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472257 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472259 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472262 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472265 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472267 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472270 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472272 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472274 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472277 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472279 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472282 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472285 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472683 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472688 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472691 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472694 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472697 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 11:03:48.495024 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472699 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472702 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472705 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472708 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472710 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472714 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472717 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472720 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472722 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472725 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472727 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472730 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472733 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472736 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472738 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472741 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472743 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472746 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472748 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472751 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 11:03:48.495854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472753 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472756 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472758 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472761 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472764 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472766 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472768 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472772 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472775 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472778 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472780 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472783 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472785 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472788 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472790 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472793 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472795 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472799 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472820 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472823 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 11:03:48.496911 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472827 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472830 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472833 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472836 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472840 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472843 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472846 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472848 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472851 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472853 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472856 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472859 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472861 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472863 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472866 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472869 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472871 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472874 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472876 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472880 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 11:03:48.497669 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472883 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472885 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472888 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472892 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472896 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472899 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472902 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472905 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472908 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472911 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472914 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472916 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472919 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472921 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472925 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472928 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472931 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472934 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472937 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 11:03:48.498317 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472939 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472942 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472944 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472947 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472950 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472954 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472959 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472962 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472965 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472968 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472970 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472973 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472977 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472980 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.472982 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473049 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473057 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473064 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473068 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473072 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473075 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 11:03:48.498874 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473079 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473085 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473088 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473091 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473094 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473098 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473101 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473104 2580 flags.go:64] FLAG: --cgroup-root="" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473107 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473110 2580 flags.go:64] FLAG: --client-ca-file="" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473114 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473117 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473121 2580 flags.go:64] FLAG: --cluster-domain="" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473123 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473127 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473130 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473133 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473137 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473140 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473143 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473146 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473149 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473152 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473156 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 11:03:48.499635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473159 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473162 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473166 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473170 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473173 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473175 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473178 2580 flags.go:64] FLAG: --enable-server="true" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473181 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473187 2580 flags.go:64] FLAG: --event-burst="100" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473191 2580 flags.go:64] FLAG: --event-qps="50" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473194 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473197 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473200 2580 flags.go:64] FLAG: --eviction-hard="" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473204 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473207 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473210 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473212 2580 flags.go:64] FLAG: --eviction-soft="" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473215 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473218 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473222 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473225 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473227 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473231 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473234 2580 flags.go:64] FLAG: --feature-gates="" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473237 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 11:03:48.500711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473240 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473243 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473246 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473249 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473252 2580 flags.go:64] FLAG: --help="false" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473255 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-142-15.ec2.internal" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473259 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473262 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473265 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473269 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473272 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473275 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473278 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473281 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473283 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473286 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473291 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473294 2580 flags.go:64] FLAG: --kube-reserved="" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473296 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473300 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473303 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473306 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473308 2580 flags.go:64] FLAG: --lock-file="" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473311 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 11:03:48.501505 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473314 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473317 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473323 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473326 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473329 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473332 2580 flags.go:64] FLAG: --logging-format="text" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473335 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473338 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473341 2580 flags.go:64] FLAG: --manifest-url="" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473344 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473348 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473351 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473355 2580 flags.go:64] FLAG: --max-pods="110" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473358 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473365 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473368 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473371 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473374 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473377 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473380 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473387 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473390 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473393 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 11:03:48.502727 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473396 2580 flags.go:64] FLAG: --pod-cidr="" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473401 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473406 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473409 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473412 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473414 2580 flags.go:64] FLAG: --port="10250" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473418 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473421 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0de4085cbb679209e" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473424 2580 flags.go:64] FLAG: --qos-reserved="" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473427 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473430 2580 flags.go:64] FLAG: --register-node="true" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473432 2580 flags.go:64] FLAG: --register-with-taints="" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473436 2580 flags.go:64] FLAG: --registry-burst="10" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473439 2580 flags.go:64] FLAG: --registry-qps="5" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473442 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473445 2580 flags.go:64] FLAG: --reserved-memory="" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473449 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473452 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473455 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473458 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473460 2580 flags.go:64] FLAG: --runonce="false" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473463 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473466 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473469 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473474 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 11:03:48.503370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473476 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473479 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473482 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473485 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473488 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473491 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473494 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473497 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473501 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473504 2580 flags.go:64] FLAG: --system-cgroups="" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473507 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473512 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473515 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473518 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473522 2580 flags.go:64] FLAG: --tls-min-version="" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473524 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473527 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473530 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473533 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473536 2580 flags.go:64] FLAG: --v="2" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473540 2580 flags.go:64] FLAG: --version="false" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473545 2580 flags.go:64] FLAG: --vmodule="" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473548 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473551 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 11:03:48.560826 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473684 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 11:03:48.549352 ip-10-0-142-15 systemd[1]: Started Kubernetes Kubelet. Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473690 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473693 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473696 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473700 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473702 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473706 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473710 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473713 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473715 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473718 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473720 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473723 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473725 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473728 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473730 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473734 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473737 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473739 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 11:03:48.562100 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473742 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473746 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473749 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473752 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473755 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473758 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473761 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473763 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473766 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473768 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473770 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473774 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473777 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473779 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473782 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473784 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473786 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473789 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473791 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473794 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 11:03:48.563456 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473797 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473800 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473803 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473805 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473808 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473810 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473813 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473815 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473818 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473822 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473825 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473827 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473830 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473833 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473835 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473837 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473840 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473842 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473845 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 11:03:48.564890 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473847 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473850 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473853 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473855 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473857 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473867 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473869 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473872 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473874 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473877 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473879 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473882 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473884 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473888 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473891 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473893 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473896 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473898 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473901 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473903 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 11:03:48.570257 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473906 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473909 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473912 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473915 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473917 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473920 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473923 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473925 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473928 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473930 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473933 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473935 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473938 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473941 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473943 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473946 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473950 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473953 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473957 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473959 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 11:03:48.571460 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.473962 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.473971 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.480893 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.480910 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.481682 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.481691 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481762 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481767 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481772 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481777 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481782 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481786 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481789 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481791 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481795 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 11:03:48.572814 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481799 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481801 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481804 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481808 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481810 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481813 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481816 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481819 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481822 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481824 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481827 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481831 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481834 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481837 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481841 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481845 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481848 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481850 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481854 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481857 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 11:03:48.573487 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481862 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481865 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481869 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481871 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481874 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481877 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481881 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481883 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481886 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481890 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481893 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481895 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481898 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481902 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481905 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481910 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481915 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481918 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481920 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 11:03:48.574868 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481924 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481928 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481930 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481933 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481936 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481939 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481942 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481944 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481948 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481953 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481957 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481962 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481965 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481968 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481971 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481974 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481978 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481982 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 11:03:48.579029 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481984 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481989 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481994 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.481997 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482000 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482004 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482006 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482009 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482012 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482016 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482020 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482023 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482026 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482030 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482032 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482035 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482038 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482042 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482044 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 11:03:48.579708 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482047 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482051 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482053 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482056 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482058 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482062 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482065 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482067 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482070 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482073 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482077 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482080 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482082 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482086 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.482091 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 11:03:48.580847 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482245 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482252 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482255 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482258 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482261 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482264 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482269 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482272 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482277 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482279 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482282 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482285 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482289 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482292 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482297 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482300 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482303 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482306 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482309 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 11:03:48.582283 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482312 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482315 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482317 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482320 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482323 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482326 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482329 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482331 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482335 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482345 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482348 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482351 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482353 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482356 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482359 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482362 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482364 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482367 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482370 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482375 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 11:03:48.582952 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482378 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482397 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482402 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482406 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482408 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482412 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482415 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482418 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482421 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482423 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482427 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482430 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482433 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482436 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482439 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482442 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482444 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482447 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482450 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482454 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 11:03:48.583876 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482457 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482460 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482469 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482472 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482476 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482478 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482481 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482484 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482487 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482490 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482493 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482497 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482500 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482505 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482507 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482510 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482514 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482517 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482521 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482525 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 11:03:48.654135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482529 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482532 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482535 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482538 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482542 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482545 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482547 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482550 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482553 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482556 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482559 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482561 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482564 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482567 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482590 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482602 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482607 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482610 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482613 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 11:03:48.655504 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:48.482616 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 11:03:48.657321 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.482622 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 11:03:48.657321 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.482753 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 11:03:48.657321 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.494677 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 11:03:48.657321 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.495983 2580 server.go:1019] "Starting client certificate rotation" Jul 16 11:03:48.657321 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.497423 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 11:03:48.657321 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.497529 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 11:03:48.657321 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.498685 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 11:03:48.657321 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.500220 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 11:03:48.657321 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.512205 2580 log.go:25] "Validated CRI v1 runtime API" Jul 16 11:03:48.657321 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.515811 2580 log.go:25] "Validated CRI v1 image API" Jul 16 11:03:48.657321 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.516658 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 11:03:48.657321 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.520867 2580 fs.go:135] Filesystem UUIDs: map[2053c96f-80d1-4253-a068-8a703cb8abe2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a1cf585d-1bf7-4af1-95ed-b2ff0ce13741:/dev/nvme0n1p3] Jul 16 11:03:48.657910 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.520886 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 11:03:48.659080 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.527729 2580 manager.go:216] Machine: {Timestamp:2026-07-16 11:03:48.525549204 +0000 UTC m=+0.358897878 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103114 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23ae1980d182ea37c631cc60b735fc SystemUUID:ec23ae19-80d1-82ea-37c6-31cc60b735fc BootID:b0467b70-98f6-4f0e-bd9e-736e7f986298 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:63:7b:c6:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:63:7b:c6:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:e9:3d:70:eb:7a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 11:03:48.659080 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.527862 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 11:03:48.659080 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.527999 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 11:03:48.659080 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.531214 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 11:03:48.659080 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.531244 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-15.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 11:03:48.659080 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.531939 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 11:03:48.659080 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.531953 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 11:03:48.659080 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.531961 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.533377 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.534677 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.537554 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.537723 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.537747 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.537759 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.537779 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.537795 2580 kubelet.go:398] "Adding apiserver pod source" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.537804 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.537852 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.538181 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.538204 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.540460 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.542737 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.542774 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.543002 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.543018 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.543026 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.543031 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.543040 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.543048 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.543518 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.543526 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.543535 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.543540 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.543550 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.543559 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.544505 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.544515 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.548163 2580 server.go:1268] "Started kubelet" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.548287 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.548457 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.548700 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.549554 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-15.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.549832 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.549942 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.550200 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.550202 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 11:03:48.659518 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.556505 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.559491 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.559508 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.559618 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.560135 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.560136 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.560161 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.560243 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.560274 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.560501 2580 factory.go:55] Registering systemd factory Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.560515 2580 factory.go:223] Registration of the systemd container factory successfully Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.560756 2580 factory.go:153] Registering CRI-O factory Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.560771 2580 factory.go:223] Registration of the crio container factory successfully Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.560819 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.560841 2580 factory.go:103] Registering Raw factory Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.560856 2580 manager.go:1196] Started watching for new ooms in manager Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.561250 2580 manager.go:349] Starting recovery of all containers Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.560249 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-15.ec2.internal.18c2c04a7ceb6597 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-15.ec2.internal,UID:ip-10-0-142-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-15.ec2.internal,},FirstTimestamp:2026-07-16 11:03:48.548052375 +0000 UTC m=+0.381401047,LastTimestamp:2026-07-16 11:03:48.548052375 +0000 UTC m=+0.381401047,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-15.ec2.internal,}" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.571489 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.571601 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.572031 2580 manager.go:354] Recovery completed Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.575392 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v2nhw" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.577731 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.580074 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.580106 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.580119 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.580655 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.580668 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.580686 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.580728 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.581732 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v2nhw" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.582484 2580 policy_none.go:49] "None policy: Start" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.582504 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.582516 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 11:03:48.660970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.583499 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:03:48.662050 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.583539 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:03:48.662050 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.583557 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:03:48.662050 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.593380 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-15.ec2.internal.18c2c04a7ed438b9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-15.ec2.internal,UID:ip-10-0-142-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-15.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-15.ec2.internal,},FirstTimestamp:2026-07-16 11:03:48.580087993 +0000 UTC m=+0.413436675,LastTimestamp:2026-07-16 11:03:48.580087993 +0000 UTC m=+0.413436675,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-15.ec2.internal,}" Jul 16 11:03:48.662050 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.595950 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-15.ec2.internal.18c2c04a7ed49832 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-15.ec2.internal,UID:ip-10-0-142-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-15.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-15.ec2.internal,},FirstTimestamp:2026-07-16 11:03:48.580112434 +0000 UTC m=+0.413461106,LastTimestamp:2026-07-16 11:03:48.580112434 +0000 UTC m=+0.413461106,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-15.ec2.internal,}" Jul 16 11:03:48.662050 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.604477 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-15.ec2.internal.18c2c04a7ed4c0fa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-15.ec2.internal,UID:ip-10-0-142-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-15.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-15.ec2.internal,},FirstTimestamp:2026-07-16 11:03:48.580122874 +0000 UTC m=+0.413471547,LastTimestamp:2026-07-16 11:03:48.580122874 +0000 UTC m=+0.413471547,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-15.ec2.internal,}" Jul 16 11:03:48.662050 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.610970 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-15.ec2.internal.18c2c04a7ed438b9\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-15.ec2.internal.18c2c04a7ed438b9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-15.ec2.internal,UID:ip-10-0-142-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-15.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-15.ec2.internal,},FirstTimestamp:2026-07-16 11:03:48.580087993 +0000 UTC m=+0.413436675,LastTimestamp:2026-07-16 11:03:48.583518176 +0000 UTC m=+0.416866855,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-15.ec2.internal,}" Jul 16 11:03:48.662293 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.615823 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-15.ec2.internal.18c2c04a7ed49832\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-15.ec2.internal.18c2c04a7ed49832 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-15.ec2.internal,UID:ip-10-0-142-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-15.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-15.ec2.internal,},FirstTimestamp:2026-07-16 11:03:48.580112434 +0000 UTC m=+0.413461106,LastTimestamp:2026-07-16 11:03:48.583545942 +0000 UTC m=+0.416894617,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-15.ec2.internal,}" Jul 16 11:03:48.662293 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.623775 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-15.ec2.internal.18c2c04a7ed4c0fa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-15.ec2.internal.18c2c04a7ed4c0fa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-15.ec2.internal,UID:ip-10-0-142-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-15.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-15.ec2.internal,},FirstTimestamp:2026-07-16 11:03:48.580122874 +0000 UTC m=+0.413471547,LastTimestamp:2026-07-16 11:03:48.583566307 +0000 UTC m=+0.416914982,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-15.ec2.internal,}" Jul 16 11:03:48.662293 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.636553 2580 policy_none.go:47] "Start" Jul 16 11:03:48.662293 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.659943 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:48.783829 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.689423 2580 manager.go:337] "Starting Device Plugin manager" Jul 16 11:03:48.783829 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.689506 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 11:03:48.783829 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.689522 2580 server.go:87] "Starting device plugin registration server" Jul 16 11:03:48.783829 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.689840 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 11:03:48.783829 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.689854 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 11:03:48.783829 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.689992 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 11:03:48.783829 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.690077 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 11:03:48.783829 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.690086 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 11:03:48.783829 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.690645 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 11:03:48.783829 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.690707 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:48.783829 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.694872 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-15.ec2.internal.18c2c04a85863354 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-15.ec2.internal,UID:ip-10-0-142-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-142-15.ec2.internal,},FirstTimestamp:2026-07-16 11:03:48.692415316 +0000 UTC m=+0.525763979,LastTimestamp:2026-07-16 11:03:48.692415316 +0000 UTC m=+0.525763979,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-15.ec2.internal,}" Jul 16 11:03:48.789270 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.788153 2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-15.ec2.internal\" not found" node="ip-10-0-142-15.ec2.internal" Jul 16 11:03:48.790664 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.790287 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:03:48.791682 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.791311 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:03:48.791682 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.791340 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:03:48.791682 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.791352 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:03:48.791682 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.791382 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-15.ec2.internal" Jul 16 11:03:48.918616 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.801264 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-15.ec2.internal" Jul 16 11:03:48.918616 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.801285 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-15.ec2.internal\": node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:48.918616 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.815827 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:48.918616 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.878494 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 11:03:48.918616 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.879830 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 11:03:48.918616 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.879872 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 11:03:48.918616 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.879898 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 11:03:48.918616 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.879934 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 11:03:48.918616 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.882235 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 11:03:48.918616 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:48.916972 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.980995 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-15.ec2.internal"] Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.981098 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.982263 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.982295 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.982307 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.983566 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.983750 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.983778 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.984428 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.984464 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.984478 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.984435 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.984540 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.984553 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.985857 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.985889 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.995000 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.995038 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:48.995055 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:49.017218 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:49.031710 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-15.ec2.internal\" not found" node="ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.042628 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:49.036543 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-15.ec2.internal\" not found" node="ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.138488 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.063102 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/85ac7ba77ee7d6a135faaf9d8c1bdae9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal\" (UID: \"85ac7ba77ee7d6a135faaf9d8c1bdae9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.138488 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.063143 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85ac7ba77ee7d6a135faaf9d8c1bdae9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal\" (UID: \"85ac7ba77ee7d6a135faaf9d8c1bdae9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.138488 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.063163 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2e3677082925fab7d36c1bc7a29637d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-15.ec2.internal\" (UID: \"b2e3677082925fab7d36c1bc7a29637d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.138488 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:49.117626 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:49.275439 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.163886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/85ac7ba77ee7d6a135faaf9d8c1bdae9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal\" (UID: \"85ac7ba77ee7d6a135faaf9d8c1bdae9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.275439 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.163976 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/85ac7ba77ee7d6a135faaf9d8c1bdae9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal\" (UID: \"85ac7ba77ee7d6a135faaf9d8c1bdae9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.275439 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.164018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85ac7ba77ee7d6a135faaf9d8c1bdae9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal\" (UID: \"85ac7ba77ee7d6a135faaf9d8c1bdae9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.275439 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.164037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2e3677082925fab7d36c1bc7a29637d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-15.ec2.internal\" (UID: \"b2e3677082925fab7d36c1bc7a29637d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.275439 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.164063 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2e3677082925fab7d36c1bc7a29637d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-15.ec2.internal\" (UID: \"b2e3677082925fab7d36c1bc7a29637d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.275439 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.164089 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85ac7ba77ee7d6a135faaf9d8c1bdae9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal\" (UID: \"85ac7ba77ee7d6a135faaf9d8c1bdae9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.275439 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:49.221628 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:49.396995 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:49.323014 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:49.396995 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.334630 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.396995 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.339993 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-15.ec2.internal" Jul 16 11:03:49.515511 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:49.426488 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:49.515511 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.497483 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 11:03:49.515511 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.497719 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 11:03:49.622583 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:49.527977 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:49.628516 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:49.628487 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:49.639733 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.639697 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 10:58:48 +0000 UTC" deadline="2028-01-14 22:03:40.891849689 +0000 UTC" Jul 16 11:03:49.639791 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.639735 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13138h59m51.252120073s" Jul 16 11:03:49.692006 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.691930 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:03:49.693203 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.693182 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:03:49.693305 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.693222 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:03:49.693305 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:49.693240 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:03:49.729647 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:49.729608 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:49.832827 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:49.832787 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:49.936558 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:49.936520 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:50.018773 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.018701 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 11:03:50.036716 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:50.036682 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-15.ec2.internal\" not found" Jul 16 11:03:50.079517 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.079491 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 11:03:50.127763 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.127730 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 11:03:50.160742 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.160716 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal" Jul 16 11:03:50.174124 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.174100 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 11:03:50.175013 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.175000 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-15.ec2.internal" Jul 16 11:03:50.186916 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.186894 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 11:03:50.415873 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:50.415832 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ac7ba77ee7d6a135faaf9d8c1bdae9.slice/crio-8956edf25827c9ca7d38657e6ea52ee4f30c9a9234cdd61ed947fc20fb671b4f WatchSource:0}: Error finding container 8956edf25827c9ca7d38657e6ea52ee4f30c9a9234cdd61ed947fc20fb671b4f: Status 404 returned error can't find the container with id 8956edf25827c9ca7d38657e6ea52ee4f30c9a9234cdd61ed947fc20fb671b4f Jul 16 11:03:50.421860 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.421841 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 11:03:50.527625 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:50.527581 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e3677082925fab7d36c1bc7a29637d.slice/crio-e11086fb529bb30827ce94d8384149d98f0049bbf04b06090fffa5c66c626383 WatchSource:0}: Error finding container e11086fb529bb30827ce94d8384149d98f0049bbf04b06090fffa5c66c626383: Status 404 returned error can't find the container with id e11086fb529bb30827ce94d8384149d98f0049bbf04b06090fffa5c66c626383 Jul 16 11:03:50.538814 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.538795 2580 apiserver.go:52] "Watching apiserver" Jul 16 11:03:50.546618 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.546555 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 11:03:50.548734 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.548710 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-dsbh6","openshift-multus/network-metrics-daemon-dvxhb","openshift-network-diagnostics/network-check-target-c7mlk","kube-system/kube-apiserver-proxy-ip-10-0-142-15.ec2.internal","openshift-cluster-node-tuning-operator/tuned-xnd7p","openshift-dns/node-resolver-ss9lt","openshift-multus/multus-95rzf","openshift-network-operator/iptables-alerter-lwzs8","openshift-ovn-kubernetes/ovnkube-node-66mkq","kube-system/global-pull-secret-syncer-fxlr6","kube-system/konnectivity-agent-mn5mk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf","openshift-image-registry/node-ca-r76fs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal"] Jul 16 11:03:50.551685 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.551665 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.553905 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.553883 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mn5mk" Jul 16 11:03:50.554589 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.554568 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 11:03:50.554712 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.554571 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 11:03:50.554712 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.554612 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zkg8w\"" Jul 16 11:03:50.556156 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.556134 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:50.556254 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:50.556214 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:03:50.556442 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.556424 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8nbfp\"" Jul 16 11:03:50.556497 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.556448 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 11:03:50.556545 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.556505 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 11:03:50.558448 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.558428 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:03:50.558539 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:50.558516 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:03:50.560510 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.560493 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 11:03:50.560765 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.560744 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ss9lt" Jul 16 11:03:50.563173 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.563151 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.563685 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.563654 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 11:03:50.563685 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.563681 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tkdh6\"" Jul 16 11:03:50.563843 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.563655 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 11:03:50.565792 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.565752 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 11:03:50.565934 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.565907 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lwzs8" Jul 16 11:03:50.566295 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.565992 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 11:03:50.566295 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.566015 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 11:03:50.566295 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.566052 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-blj22\"" Jul 16 11:03:50.566295 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.566104 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 11:03:50.568571 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.568551 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.568695 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.568590 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 11:03:50.568765 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.568714 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xbjqj\"" Jul 16 11:03:50.569198 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.569176 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 11:03:50.569276 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.569262 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 11:03:50.571040 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571021 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-tuned\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.571133 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2rjr\" (UniqueName: \"kubernetes.io/projected/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-kube-api-access-p2rjr\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.571133 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571084 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-modprobe-d\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.571133 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571110 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-kubernetes\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.571298 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-tmp\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.571298 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e5ba1b3-7981-43d4-b14a-9ace2f2e1128-agent-certs\") pod \"konnectivity-agent-mn5mk\" (UID: \"2e5ba1b3-7981-43d4-b14a-9ace2f2e1128\") " pod="kube-system/konnectivity-agent-mn5mk" Jul 16 11:03:50.571298 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571199 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25a48584-b83e-4cff-a68f-18c383e81fc7-kubelet-config\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:50.571298 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25a48584-b83e-4cff-a68f-18c383e81fc7-dbus\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:50.571298 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571264 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 11:03:50.571298 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w6bg\" (UniqueName: \"kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg\") pod \"network-check-target-c7mlk\" (UID: \"4055addb-97de-4ae7-b8e3-296404d93869\") " pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:03:50.571600 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e5ba1b3-7981-43d4-b14a-9ace2f2e1128-konnectivity-ca\") pod \"konnectivity-agent-mn5mk\" (UID: \"2e5ba1b3-7981-43d4-b14a-9ace2f2e1128\") " pod="kube-system/konnectivity-agent-mn5mk" Jul 16 11:03:50.571600 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:50.571600 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571373 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/daaff1e0-e805-446f-8a08-4baf8c77191d-hosts-file\") pod \"node-resolver-ss9lt\" (UID: \"daaff1e0-e805-446f-8a08-4baf8c77191d\") " pod="openshift-dns/node-resolver-ss9lt" Jul 16 11:03:50.571600 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daaff1e0-e805-446f-8a08-4baf8c77191d-tmp-dir\") pod \"node-resolver-ss9lt\" (UID: \"daaff1e0-e805-446f-8a08-4baf8c77191d\") " pod="openshift-dns/node-resolver-ss9lt" Jul 16 11:03:50.571600 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571437 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdj5l\" (UniqueName: \"kubernetes.io/projected/daaff1e0-e805-446f-8a08-4baf8c77191d-kube-api-access-hdj5l\") pod \"node-resolver-ss9lt\" (UID: \"daaff1e0-e805-446f-8a08-4baf8c77191d\") " pod="openshift-dns/node-resolver-ss9lt" Jul 16 11:03:50.571600 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571468 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-sysctl-d\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.571600 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571476 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 11:03:50.571600 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571479 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 11:03:50.571600 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-run\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.571600 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-sys\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.571600 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-host\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.572182 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.572182 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-sysconfig\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.572182 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571698 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-sysctl-conf\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.572182 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571739 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-systemd\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.572182 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-lib-modules\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.572182 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.571786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-var-lib-kubelet\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.572479 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.572281 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 11:03:50.572479 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.572344 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 11:03:50.572479 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.572369 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 11:03:50.572479 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.572439 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nbc8v\"" Jul 16 11:03:50.575145 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.574554 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 11:03:50.575145 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.574699 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 11:03:50.575145 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.574737 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4wd59\"" Jul 16 11:03:50.575466 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.575440 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:03:50.575691 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:50.575657 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:03:50.577002 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.576978 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 11:03:50.578574 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.578554 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.580941 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.580922 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 11:03:50.581007 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.580985 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r76fs" Jul 16 11:03:50.581184 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.581159 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 11:03:50.581290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.581265 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dzhr8\"" Jul 16 11:03:50.581389 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.581269 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 11:03:50.583421 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.583397 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 11:03:50.583609 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.583594 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2sjgd\"" Jul 16 11:03:50.583609 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.583604 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 11:03:50.583756 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.583620 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 11:03:50.620468 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.620440 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8dl7b" Jul 16 11:03:50.631273 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.631250 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8dl7b" Jul 16 11:03:50.661344 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.661313 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 11:03:50.671969 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.671938 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-run-multus-certs\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.672064 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.671984 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmrrf\" (UniqueName: \"kubernetes.io/projected/870684c4-b8b2-48d5-8c2f-97c635fad4df-kube-api-access-bmrrf\") pod \"node-ca-r76fs\" (UID: \"870684c4-b8b2-48d5-8c2f-97c635fad4df\") " pod="openshift-image-registry/node-ca-r76fs" Jul 16 11:03:50.672064 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-node-log\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.672064 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-log-socket\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.672064 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672039 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/04bb84b9-bd39-419b-947a-2ab4c9d0476d-ovnkube-config\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.672064 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-tuned\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.672296 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/870684c4-b8b2-48d5-8c2f-97c635fad4df-host\") pod \"node-ca-r76fs\" (UID: \"870684c4-b8b2-48d5-8c2f-97c635fad4df\") " pod="openshift-image-registry/node-ca-r76fs" Jul 16 11:03:50.672296 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672104 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/870684c4-b8b2-48d5-8c2f-97c635fad4df-serviceca\") pod \"node-ca-r76fs\" (UID: \"870684c4-b8b2-48d5-8c2f-97c635fad4df\") " pod="openshift-image-registry/node-ca-r76fs" Jul 16 11:03:50.672296 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672119 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-var-lib-openvswitch\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.672296 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/daaff1e0-e805-446f-8a08-4baf8c77191d-hosts-file\") pod \"node-resolver-ss9lt\" (UID: \"daaff1e0-e805-446f-8a08-4baf8c77191d\") " pod="openshift-dns/node-resolver-ss9lt" Jul 16 11:03:50.672296 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61cc2f50-0bdd-400e-92da-09235b2d344b-cni-binary-copy\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.672296 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8mc5\" (UniqueName: \"kubernetes.io/projected/61cc2f50-0bdd-400e-92da-09235b2d344b-kube-api-access-v8mc5\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.672296 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e5ba1b3-7981-43d4-b14a-9ace2f2e1128-konnectivity-ca\") pod \"konnectivity-agent-mn5mk\" (UID: \"2e5ba1b3-7981-43d4-b14a-9ace2f2e1128\") " pod="kube-system/konnectivity-agent-mn5mk" Jul 16 11:03:50.672296 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daaff1e0-e805-446f-8a08-4baf8c77191d-tmp-dir\") pod \"node-resolver-ss9lt\" (UID: \"daaff1e0-e805-446f-8a08-4baf8c77191d\") " pod="openshift-dns/node-resolver-ss9lt" Jul 16 11:03:50.672296 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-var-lib-cni-multus\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.672296 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672249 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/daaff1e0-e805-446f-8a08-4baf8c77191d-hosts-file\") pod \"node-resolver-ss9lt\" (UID: \"daaff1e0-e805-446f-8a08-4baf8c77191d\") " pod="openshift-dns/node-resolver-ss9lt" Jul 16 11:03:50.672296 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672286 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/04bb84b9-bd39-419b-947a-2ab4c9d0476d-ovn-node-metrics-cert\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.672882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-host\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.672882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-var-lib-kubelet\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.672882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7e80ccb-f7c2-469c-b52c-179371d3da15-system-cni-dir\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.672882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672453 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9wh5\" (UniqueName: \"kubernetes.io/projected/d7e80ccb-f7c2-469c-b52c-179371d3da15-kube-api-access-m9wh5\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.672882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672459 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-host\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.672882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672454 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 11:03:50.672882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672485 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7e80ccb-f7c2-469c-b52c-179371d3da15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.672882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e5ba1b3-7981-43d4-b14a-9ace2f2e1128-agent-certs\") pod \"konnectivity-agent-mn5mk\" (UID: \"2e5ba1b3-7981-43d4-b14a-9ace2f2e1128\") " pod="kube-system/konnectivity-agent-mn5mk" Jul 16 11:03:50.672882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daaff1e0-e805-446f-8a08-4baf8c77191d-tmp-dir\") pod \"node-resolver-ss9lt\" (UID: \"daaff1e0-e805-446f-8a08-4baf8c77191d\") " pod="openshift-dns/node-resolver-ss9lt" Jul 16 11:03:50.672882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-multus-conf-dir\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.672882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25a48584-b83e-4cff-a68f-18c383e81fc7-dbus\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:50.672882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-cnibin\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.672882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e5ba1b3-7981-43d4-b14a-9ace2f2e1128-konnectivity-ca\") pod \"konnectivity-agent-mn5mk\" (UID: \"2e5ba1b3-7981-43d4-b14a-9ace2f2e1128\") " pod="kube-system/konnectivity-agent-mn5mk" Jul 16 11:03:50.672882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672870 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25a48584-b83e-4cff-a68f-18c383e81fc7-dbus\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:50.673483 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672840 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9a9276-10aa-4cee-844f-46bc4494697b-host-slash\") pod \"iptables-alerter-lwzs8\" (UID: \"eb9a9276-10aa-4cee-844f-46bc4494697b\") " pod="openshift-network-operator/iptables-alerter-lwzs8" Jul 16 11:03:50.673483 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7e80ccb-f7c2-469c-b52c-179371d3da15-cnibin\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.673483 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7e80ccb-f7c2-469c-b52c-179371d3da15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.673483 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-etc-openvswitch\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.673483 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.672991 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/04bb84b9-bd39-419b-947a-2ab4c9d0476d-env-overrides\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.673483 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673014 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-system-cni-dir\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.673483 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673071 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-multus-socket-dir-parent\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.673483 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-hostroot\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.673483 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwthz\" (UniqueName: \"kubernetes.io/projected/eb9a9276-10aa-4cee-844f-46bc4494697b-kube-api-access-wwthz\") pod \"iptables-alerter-lwzs8\" (UID: \"eb9a9276-10aa-4cee-844f-46bc4494697b\") " pod="openshift-network-operator/iptables-alerter-lwzs8" Jul 16 11:03:50.673483 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673163 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7e80ccb-f7c2-469c-b52c-179371d3da15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.673483 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-registration-dir\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.673483 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-kubelet\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.673483 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-cni-bin\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.674056 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673259 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-os-release\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.674056 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673292 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7e80ccb-f7c2-469c-b52c-179371d3da15-cni-binary-copy\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.674056 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673328 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-slash\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.674056 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673348 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.674056 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk987\" (UniqueName: \"kubernetes.io/projected/2e52bd54-2b39-43e6-8494-8b74da851f0f-kube-api-access-xk987\") pod \"network-metrics-daemon-dvxhb\" (UID: \"2e52bd54-2b39-43e6-8494-8b74da851f0f\") " pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:03:50.674056 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-systemd\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.674056 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-lib-modules\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.674056 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673480 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-systemd\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.674056 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673527 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rjr\" (UniqueName: \"kubernetes.io/projected/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-kube-api-access-p2rjr\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.674056 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673610 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7e80ccb-f7c2-469c-b52c-179371d3da15-os-release\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.674056 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673650 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.674056 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpcr2\" (UniqueName: \"kubernetes.io/projected/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-kube-api-access-zpcr2\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.674056 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673697 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-systemd-units\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.674056 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673701 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-lib-modules\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.674731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/04bb84b9-bd39-419b-947a-2ab4c9d0476d-ovnkube-script-lib\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.674731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-modprobe-d\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.674731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673821 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-kubernetes\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.674731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673846 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-modprobe-d\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.674731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-tmp\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.674731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673895 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-kubernetes\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.674731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.673964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w6bg\" (UniqueName: \"kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg\") pod \"network-check-target-c7mlk\" (UID: \"4055addb-97de-4ae7-b8e3-296404d93869\") " pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:03:50.674731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdj5l\" (UniqueName: \"kubernetes.io/projected/daaff1e0-e805-446f-8a08-4baf8c77191d-kube-api-access-hdj5l\") pod \"node-resolver-ss9lt\" (UID: \"daaff1e0-e805-446f-8a08-4baf8c77191d\") " pod="openshift-dns/node-resolver-ss9lt" Jul 16 11:03:50.674731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674116 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-run-openvswitch\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.674731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd2n5\" (UniqueName: \"kubernetes.io/projected/04bb84b9-bd39-419b-947a-2ab4c9d0476d-kube-api-access-wd2n5\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.674731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:50.674731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-sysctl-d\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.674731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-run\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.674731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-sys\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.675275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-run-netns\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.675275 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:50.674379 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 11:03:50.675275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674393 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-sys-fs\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.675275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674402 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-sys\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.675275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674404 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-run\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.675275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674432 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-sysctl-d\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.675275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674417 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-run-systemd\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.675275 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:50.674471 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret podName:25a48584-b83e-4cff-a68f-18c383e81fc7 nodeName:}" failed. No retries permitted until 2026-07-16 11:03:51.17442827 +0000 UTC m=+3.007776954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret") pod "global-pull-secret-syncer-fxlr6" (UID: "25a48584-b83e-4cff-a68f-18c383e81fc7") : object "kube-system"/"original-pull-secret" not registered Jul 16 11:03:50.675275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-cni-netd\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.675275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674541 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs\") pod \"network-metrics-daemon-dvxhb\" (UID: \"2e52bd54-2b39-43e6-8494-8b74da851f0f\") " pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:03:50.675275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61cc2f50-0bdd-400e-92da-09235b2d344b-multus-daemon-config\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.675275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-etc-kubernetes\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.675275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674656 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-socket-dir\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.675275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674686 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25a48584-b83e-4cff-a68f-18c383e81fc7-kubelet-config\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:50.675749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-sysconfig\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.675749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-sysctl-conf\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.675749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-var-lib-kubelet\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.675749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674784 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25a48584-b83e-4cff-a68f-18c383e81fc7-kubelet-config\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:50.675749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674806 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb9a9276-10aa-4cee-844f-46bc4494697b-iptables-alerter-script\") pod \"iptables-alerter-lwzs8\" (UID: \"eb9a9276-10aa-4cee-844f-46bc4494697b\") " pod="openshift-network-operator/iptables-alerter-lwzs8" Jul 16 11:03:50.675749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674814 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-sysconfig\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.675749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674833 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-var-lib-cni-bin\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.675749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-device-dir\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.675749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674870 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-var-lib-kubelet\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.675749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-run-netns\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.675749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674914 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-sysctl-conf\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.675749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674918 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-run-ovn-kubernetes\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.675749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-multus-cni-dir\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.675749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.674988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-run-k8s-cni-cncf-io\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.676270 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.675025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.676270 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.675049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-run-ovn\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.676270 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.676252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-tmp\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.676408 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.676348 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-etc-tuned\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.676575 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.676552 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e5ba1b3-7981-43d4-b14a-9ace2f2e1128-agent-certs\") pod \"konnectivity-agent-mn5mk\" (UID: \"2e5ba1b3-7981-43d4-b14a-9ace2f2e1128\") " pod="kube-system/konnectivity-agent-mn5mk" Jul 16 11:03:50.677801 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.677785 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 11:03:50.682614 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.682592 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2rjr\" (UniqueName: \"kubernetes.io/projected/e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0-kube-api-access-p2rjr\") pod \"tuned-xnd7p\" (UID: \"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0\") " pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.690476 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:50.690456 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 11:03:50.690562 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:50.690483 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 11:03:50.690562 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:50.690497 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8w6bg for pod openshift-network-diagnostics/network-check-target-c7mlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:03:50.690562 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:50.690557 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg podName:4055addb-97de-4ae7-b8e3-296404d93869 nodeName:}" failed. No retries permitted until 2026-07-16 11:03:51.190544409 +0000 UTC m=+3.023893083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8w6bg" (UniqueName: "kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg") pod "network-check-target-c7mlk" (UID: "4055addb-97de-4ae7-b8e3-296404d93869") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:03:50.692688 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.692663 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdj5l\" (UniqueName: \"kubernetes.io/projected/daaff1e0-e805-446f-8a08-4baf8c77191d-kube-api-access-hdj5l\") pod \"node-resolver-ss9lt\" (UID: \"daaff1e0-e805-446f-8a08-4baf8c77191d\") " pod="openshift-dns/node-resolver-ss9lt" Jul 16 11:03:50.776143 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-run-netns\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.776143 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-sys-fs\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.776395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-run-systemd\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.776395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-cni-netd\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.776395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776220 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-run-systemd\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.776395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776236 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-run-netns\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.776395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776246 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs\") pod \"network-metrics-daemon-dvxhb\" (UID: \"2e52bd54-2b39-43e6-8494-8b74da851f0f\") " pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:03:50.776395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-sys-fs\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.776395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61cc2f50-0bdd-400e-92da-09235b2d344b-multus-daemon-config\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.776395 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:50.776308 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:03:50.776395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-etc-kubernetes\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.776395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776328 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-cni-netd\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.776395 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:50.776352 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs podName:2e52bd54-2b39-43e6-8494-8b74da851f0f nodeName:}" failed. No retries permitted until 2026-07-16 11:03:51.276337945 +0000 UTC m=+3.109686605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs") pod "network-metrics-daemon-dvxhb" (UID: "2e52bd54-2b39-43e6-8494-8b74da851f0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:03:50.776395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-socket-dir\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.776395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776368 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-etc-kubernetes\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.776395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776386 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb9a9276-10aa-4cee-844f-46bc4494697b-iptables-alerter-script\") pod \"iptables-alerter-lwzs8\" (UID: \"eb9a9276-10aa-4cee-844f-46bc4494697b\") " pod="openshift-network-operator/iptables-alerter-lwzs8" Jul 16 11:03:50.777049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-var-lib-cni-bin\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.777049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776428 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-device-dir\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.777049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776444 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-run-netns\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.777049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-run-ovn-kubernetes\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.777049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776499 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-socket-dir\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.777049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776516 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-run-ovn-kubernetes\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.777049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776501 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-var-lib-cni-bin\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.777049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776519 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-device-dir\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.777049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-multus-cni-dir\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.777049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776545 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-run-netns\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.777049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-run-k8s-cni-cncf-io\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.777049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.777049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776650 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-multus-cni-dir\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.777049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776658 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-run-k8s-cni-cncf-io\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-run-ovn\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776698 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-run-ovn\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-run-multus-certs\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmrrf\" (UniqueName: \"kubernetes.io/projected/870684c4-b8b2-48d5-8c2f-97c635fad4df-kube-api-access-bmrrf\") pod \"node-ca-r76fs\" (UID: \"870684c4-b8b2-48d5-8c2f-97c635fad4df\") " pod="openshift-image-registry/node-ca-r76fs" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-run-multus-certs\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-node-log\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-log-socket\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/04bb84b9-bd39-419b-947a-2ab4c9d0476d-ovnkube-config\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/870684c4-b8b2-48d5-8c2f-97c635fad4df-host\") pod \"node-ca-r76fs\" (UID: \"870684c4-b8b2-48d5-8c2f-97c635fad4df\") " pod="openshift-image-registry/node-ca-r76fs" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-node-log\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/870684c4-b8b2-48d5-8c2f-97c635fad4df-serviceca\") pod \"node-ca-r76fs\" (UID: \"870684c4-b8b2-48d5-8c2f-97c635fad4df\") " pod="openshift-image-registry/node-ca-r76fs" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-var-lib-openvswitch\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776882 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-log-socket\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.777565 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776945 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-var-lib-openvswitch\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.776980 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/870684c4-b8b2-48d5-8c2f-97c635fad4df-host\") pod \"node-ca-r76fs\" (UID: \"870684c4-b8b2-48d5-8c2f-97c635fad4df\") " pod="openshift-image-registry/node-ca-r76fs" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61cc2f50-0bdd-400e-92da-09235b2d344b-cni-binary-copy\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777036 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8mc5\" (UniqueName: \"kubernetes.io/projected/61cc2f50-0bdd-400e-92da-09235b2d344b-kube-api-access-v8mc5\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777040 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61cc2f50-0bdd-400e-92da-09235b2d344b-multus-daemon-config\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-var-lib-cni-multus\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb9a9276-10aa-4cee-844f-46bc4494697b-iptables-alerter-script\") pod \"iptables-alerter-lwzs8\" (UID: \"eb9a9276-10aa-4cee-844f-46bc4494697b\") " pod="openshift-network-operator/iptables-alerter-lwzs8" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/04bb84b9-bd39-419b-947a-2ab4c9d0476d-ovn-node-metrics-cert\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-var-lib-kubelet\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7e80ccb-f7c2-469c-b52c-179371d3da15-system-cni-dir\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777189 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-var-lib-cni-multus\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wh5\" (UniqueName: \"kubernetes.io/projected/d7e80ccb-f7c2-469c-b52c-179371d3da15-kube-api-access-m9wh5\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777230 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7e80ccb-f7c2-469c-b52c-179371d3da15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777249 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-host-var-lib-kubelet\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-multus-conf-dir\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.778384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-cnibin\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9a9276-10aa-4cee-844f-46bc4494697b-host-slash\") pod \"iptables-alerter-lwzs8\" (UID: \"eb9a9276-10aa-4cee-844f-46bc4494697b\") " pod="openshift-network-operator/iptables-alerter-lwzs8" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777321 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7e80ccb-f7c2-469c-b52c-179371d3da15-system-cni-dir\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7e80ccb-f7c2-469c-b52c-179371d3da15-cnibin\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777364 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7e80ccb-f7c2-469c-b52c-179371d3da15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777372 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-multus-conf-dir\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-etc-openvswitch\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/04bb84b9-bd39-419b-947a-2ab4c9d0476d-env-overrides\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-system-cni-dir\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777442 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/870684c4-b8b2-48d5-8c2f-97c635fad4df-serviceca\") pod \"node-ca-r76fs\" (UID: \"870684c4-b8b2-48d5-8c2f-97c635fad4df\") " pod="openshift-image-registry/node-ca-r76fs" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-multus-socket-dir-parent\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-hostroot\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7e80ccb-f7c2-469c-b52c-179371d3da15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777505 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61cc2f50-0bdd-400e-92da-09235b2d344b-cni-binary-copy\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwthz\" (UniqueName: \"kubernetes.io/projected/eb9a9276-10aa-4cee-844f-46bc4494697b-kube-api-access-wwthz\") pod \"iptables-alerter-lwzs8\" (UID: \"eb9a9276-10aa-4cee-844f-46bc4494697b\") " pod="openshift-network-operator/iptables-alerter-lwzs8" Jul 16 11:03:50.779290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777554 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-hostroot\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.779953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-multus-socket-dir-parent\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.779953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7e80ccb-f7c2-469c-b52c-179371d3da15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.779953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777613 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-system-cni-dir\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.779953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-registration-dir\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.779953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-registration-dir\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.779953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-kubelet\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.779953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-cni-bin\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.779953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777756 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-os-release\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.779953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7e80ccb-f7c2-469c-b52c-179371d3da15-cni-binary-copy\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.779953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777804 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-slash\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.779953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777817 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-etc-openvswitch\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.779953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.779953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777833 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-cni-bin\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.779953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk987\" (UniqueName: \"kubernetes.io/projected/2e52bd54-2b39-43e6-8494-8b74da851f0f-kube-api-access-xk987\") pod \"network-metrics-daemon-dvxhb\" (UID: \"2e52bd54-2b39-43e6-8494-8b74da851f0f\") " pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:03:50.780395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-kubelet\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.780395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777888 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9a9276-10aa-4cee-844f-46bc4494697b-host-slash\") pod \"iptables-alerter-lwzs8\" (UID: \"eb9a9276-10aa-4cee-844f-46bc4494697b\") " pod="openshift-network-operator/iptables-alerter-lwzs8" Jul 16 11:03:50.780395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7e80ccb-f7c2-469c-b52c-179371d3da15-os-release\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.780395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.780395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpcr2\" (UniqueName: \"kubernetes.io/projected/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-kube-api-access-zpcr2\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.780395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777976 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7e80ccb-f7c2-469c-b52c-179371d3da15-cnibin\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.780395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.777990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-systemd-units\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.780395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778017 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/04bb84b9-bd39-419b-947a-2ab4c9d0476d-ovnkube-script-lib\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.780395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778018 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7e80ccb-f7c2-469c-b52c-179371d3da15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.780395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778018 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7e80ccb-f7c2-469c-b52c-179371d3da15-os-release\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.780395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-os-release\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.780395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778075 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-run-openvswitch\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.780395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778084 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/04bb84b9-bd39-419b-947a-2ab4c9d0476d-ovnkube-config\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.780395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61cc2f50-0bdd-400e-92da-09235b2d344b-cnibin\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.780815 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778139 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-run-openvswitch\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.780815 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.780815 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-slash\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.780815 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-systemd-units\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.780815 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd2n5\" (UniqueName: \"kubernetes.io/projected/04bb84b9-bd39-419b-947a-2ab4c9d0476d-kube-api-access-wd2n5\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.780815 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778352 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7e80ccb-f7c2-469c-b52c-179371d3da15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.780815 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778356 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bb84b9-bd39-419b-947a-2ab4c9d0476d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.780815 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778637 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7e80ccb-f7c2-469c-b52c-179371d3da15-cni-binary-copy\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.780815 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778643 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/04bb84b9-bd39-419b-947a-2ab4c9d0476d-ovnkube-script-lib\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.780815 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.778819 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/04bb84b9-bd39-419b-947a-2ab4c9d0476d-env-overrides\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.780815 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.780077 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/04bb84b9-bd39-419b-947a-2ab4c9d0476d-ovn-node-metrics-cert\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.788322 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.788293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmrrf\" (UniqueName: \"kubernetes.io/projected/870684c4-b8b2-48d5-8c2f-97c635fad4df-kube-api-access-bmrrf\") pod \"node-ca-r76fs\" (UID: \"870684c4-b8b2-48d5-8c2f-97c635fad4df\") " pod="openshift-image-registry/node-ca-r76fs" Jul 16 11:03:50.788728 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.788704 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8mc5\" (UniqueName: \"kubernetes.io/projected/61cc2f50-0bdd-400e-92da-09235b2d344b-kube-api-access-v8mc5\") pod \"multus-95rzf\" (UID: \"61cc2f50-0bdd-400e-92da-09235b2d344b\") " pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.788861 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.788780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9wh5\" (UniqueName: \"kubernetes.io/projected/d7e80ccb-f7c2-469c-b52c-179371d3da15-kube-api-access-m9wh5\") pod \"multus-additional-cni-plugins-dsbh6\" (UID: \"d7e80ccb-f7c2-469c-b52c-179371d3da15\") " pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.790895 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.790871 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd2n5\" (UniqueName: \"kubernetes.io/projected/04bb84b9-bd39-419b-947a-2ab4c9d0476d-kube-api-access-wd2n5\") pod \"ovnkube-node-66mkq\" (UID: \"04bb84b9-bd39-419b-947a-2ab4c9d0476d\") " pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.791019 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.790940 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwthz\" (UniqueName: \"kubernetes.io/projected/eb9a9276-10aa-4cee-844f-46bc4494697b-kube-api-access-wwthz\") pod \"iptables-alerter-lwzs8\" (UID: \"eb9a9276-10aa-4cee-844f-46bc4494697b\") " pod="openshift-network-operator/iptables-alerter-lwzs8" Jul 16 11:03:50.791200 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.791175 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpcr2\" (UniqueName: \"kubernetes.io/projected/450893ed-93e8-4a5d-a9b2-3e71ce176a1e-kube-api-access-zpcr2\") pod \"aws-ebs-csi-driver-node-26bwf\" (UID: \"450893ed-93e8-4a5d-a9b2-3e71ce176a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.791316 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.791302 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk987\" (UniqueName: \"kubernetes.io/projected/2e52bd54-2b39-43e6-8494-8b74da851f0f-kube-api-access-xk987\") pod \"network-metrics-daemon-dvxhb\" (UID: \"2e52bd54-2b39-43e6-8494-8b74da851f0f\") " pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:03:50.861086 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.860995 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" Jul 16 11:03:50.867782 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.867760 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mn5mk" Jul 16 11:03:50.871123 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:50.871098 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f21bc6_71e7_4101_8f32_f2c0b0ccaca0.slice/crio-313a80f8b69c7bcfcb8c6371afc2351061f8a017cc2270fc4228efd775d60b1d WatchSource:0}: Error finding container 313a80f8b69c7bcfcb8c6371afc2351061f8a017cc2270fc4228efd775d60b1d: Status 404 returned error can't find the container with id 313a80f8b69c7bcfcb8c6371afc2351061f8a017cc2270fc4228efd775d60b1d Jul 16 11:03:50.876177 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:50.876154 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5ba1b3_7981_43d4_b14a_9ace2f2e1128.slice/crio-7877c0d8d450f2c80a168218f128fec150d50f4009c53f11637102df724f9245 WatchSource:0}: Error finding container 7877c0d8d450f2c80a168218f128fec150d50f4009c53f11637102df724f9245: Status 404 returned error can't find the container with id 7877c0d8d450f2c80a168218f128fec150d50f4009c53f11637102df724f9245 Jul 16 11:03:50.877881 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.877838 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ss9lt" Jul 16 11:03:50.883039 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.883020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-95rzf" Jul 16 11:03:50.884302 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.884253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mn5mk" event={"ID":"2e5ba1b3-7981-43d4-b14a-9ace2f2e1128","Type":"ContainerStarted","Data":"7877c0d8d450f2c80a168218f128fec150d50f4009c53f11637102df724f9245"} Jul 16 11:03:50.885487 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.885457 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" event={"ID":"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0","Type":"ContainerStarted","Data":"313a80f8b69c7bcfcb8c6371afc2351061f8a017cc2270fc4228efd775d60b1d"} Jul 16 11:03:50.886813 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.886787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-15.ec2.internal" event={"ID":"b2e3677082925fab7d36c1bc7a29637d","Type":"ContainerStarted","Data":"e11086fb529bb30827ce94d8384149d98f0049bbf04b06090fffa5c66c626383"} Jul 16 11:03:50.887854 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.887833 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal" event={"ID":"85ac7ba77ee7d6a135faaf9d8c1bdae9","Type":"ContainerStarted","Data":"8956edf25827c9ca7d38657e6ea52ee4f30c9a9234cdd61ed947fc20fb671b4f"} Jul 16 11:03:50.888727 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:50.888702 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaaff1e0_e805_446f_8a08_4baf8c77191d.slice/crio-8c6fac3ca9292ec60293084daf2485d6355b63b7e278e10636157123522cd969 WatchSource:0}: Error finding container 8c6fac3ca9292ec60293084daf2485d6355b63b7e278e10636157123522cd969: Status 404 returned error can't find the container with id 8c6fac3ca9292ec60293084daf2485d6355b63b7e278e10636157123522cd969 Jul 16 11:03:50.889438 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.889419 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lwzs8" Jul 16 11:03:50.892854 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:50.892829 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cc2f50_0bdd_400e_92da_09235b2d344b.slice/crio-8b1a3885b88bf78d88e32c9d888589cfd07d0a49b9bedf3889e152e0fc791e7b WatchSource:0}: Error finding container 8b1a3885b88bf78d88e32c9d888589cfd07d0a49b9bedf3889e152e0fc791e7b: Status 404 returned error can't find the container with id 8b1a3885b88bf78d88e32c9d888589cfd07d0a49b9bedf3889e152e0fc791e7b Jul 16 11:03:50.897451 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.896476 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:03:50.901264 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.901242 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dsbh6" Jul 16 11:03:50.901411 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:50.901382 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9a9276_10aa_4cee_844f_46bc4494697b.slice/crio-628f27e8c2a40041558002c04cb7864add50abbf465f7ce748189af821e6727e WatchSource:0}: Error finding container 628f27e8c2a40041558002c04cb7864add50abbf465f7ce748189af821e6727e: Status 404 returned error can't find the container with id 628f27e8c2a40041558002c04cb7864add50abbf465f7ce748189af821e6727e Jul 16 11:03:50.907082 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.907058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" Jul 16 11:03:50.907355 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:50.907317 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bb84b9_bd39_419b_947a_2ab4c9d0476d.slice/crio-deab0c36516bef5c965ee43e9d90b1c17621ce0fdf82d8873722fa5b9ac68600 WatchSource:0}: Error finding container deab0c36516bef5c965ee43e9d90b1c17621ce0fdf82d8873722fa5b9ac68600: Status 404 returned error can't find the container with id deab0c36516bef5c965ee43e9d90b1c17621ce0fdf82d8873722fa5b9ac68600 Jul 16 11:03:50.911558 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:50.911534 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r76fs" Jul 16 11:03:50.912415 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:50.912392 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e80ccb_f7c2_469c_b52c_179371d3da15.slice/crio-d443290422168fa2adf7d095b3ebdeec5a26aafe00adc1a3fbc9c8605b1b8ded WatchSource:0}: Error finding container d443290422168fa2adf7d095b3ebdeec5a26aafe00adc1a3fbc9c8605b1b8ded: Status 404 returned error can't find the container with id d443290422168fa2adf7d095b3ebdeec5a26aafe00adc1a3fbc9c8605b1b8ded Jul 16 11:03:50.919244 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:50.919217 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450893ed_93e8_4a5d_a9b2_3e71ce176a1e.slice/crio-e770bb54fc1b1b388f98a3fe835297ba0dcfb70a92bcbcf071d11a24ae87ebb3 WatchSource:0}: Error finding container e770bb54fc1b1b388f98a3fe835297ba0dcfb70a92bcbcf071d11a24ae87ebb3: Status 404 returned error can't find the container with id e770bb54fc1b1b388f98a3fe835297ba0dcfb70a92bcbcf071d11a24ae87ebb3 Jul 16 11:03:50.926381 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:03:50.926357 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870684c4_b8b2_48d5_8c2f_97c635fad4df.slice/crio-7d77342161b065c4b81acf2d518a9c88279be43fa9737a1508bfe302cf56804f WatchSource:0}: Error finding container 7d77342161b065c4b81acf2d518a9c88279be43fa9737a1508bfe302cf56804f: Status 404 returned error can't find the container with id 7d77342161b065c4b81acf2d518a9c88279be43fa9737a1508bfe302cf56804f Jul 16 11:03:51.182129 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:51.182031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:51.182297 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:51.182229 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 11:03:51.182374 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:51.182316 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret podName:25a48584-b83e-4cff-a68f-18c383e81fc7 nodeName:}" failed. No retries permitted until 2026-07-16 11:03:52.18229008 +0000 UTC m=+4.015638747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret") pod "global-pull-secret-syncer-fxlr6" (UID: "25a48584-b83e-4cff-a68f-18c383e81fc7") : object "kube-system"/"original-pull-secret" not registered Jul 16 11:03:51.285131 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:51.283120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs\") pod \"network-metrics-daemon-dvxhb\" (UID: \"2e52bd54-2b39-43e6-8494-8b74da851f0f\") " pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:03:51.285131 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:51.283220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w6bg\" (UniqueName: \"kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg\") pod \"network-check-target-c7mlk\" (UID: \"4055addb-97de-4ae7-b8e3-296404d93869\") " pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:03:51.285131 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:51.283285 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:03:51.285131 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:51.283345 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 11:03:51.285131 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:51.283353 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs podName:2e52bd54-2b39-43e6-8494-8b74da851f0f nodeName:}" failed. No retries permitted until 2026-07-16 11:03:52.283334235 +0000 UTC m=+4.116682916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs") pod "network-metrics-daemon-dvxhb" (UID: "2e52bd54-2b39-43e6-8494-8b74da851f0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:03:51.285131 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:51.283363 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 11:03:51.285131 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:51.283376 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8w6bg for pod openshift-network-diagnostics/network-check-target-c7mlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:03:51.285131 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:51.283434 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg podName:4055addb-97de-4ae7-b8e3-296404d93869 nodeName:}" failed. No retries permitted until 2026-07-16 11:03:52.283422761 +0000 UTC m=+4.116771426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8w6bg" (UniqueName: "kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg") pod "network-check-target-c7mlk" (UID: "4055addb-97de-4ae7-b8e3-296404d93869") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:03:51.634852 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:51.634727 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 10:58:50 +0000 UTC" deadline="2028-04-21 10:35:45.163354708 +0000 UTC" Jul 16 11:03:51.634852 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:51.634770 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15479h31m53.528589581s" Jul 16 11:03:51.880367 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:51.880328 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:51.880554 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:51.880472 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:03:51.895171 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:51.895088 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" event={"ID":"04bb84b9-bd39-419b-947a-2ab4c9d0476d","Type":"ContainerStarted","Data":"deab0c36516bef5c965ee43e9d90b1c17621ce0fdf82d8873722fa5b9ac68600"} Jul 16 11:03:51.897884 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:51.897850 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-95rzf" event={"ID":"61cc2f50-0bdd-400e-92da-09235b2d344b","Type":"ContainerStarted","Data":"8b1a3885b88bf78d88e32c9d888589cfd07d0a49b9bedf3889e152e0fc791e7b"} Jul 16 11:03:51.899320 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:51.899289 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lwzs8" event={"ID":"eb9a9276-10aa-4cee-844f-46bc4494697b","Type":"ContainerStarted","Data":"628f27e8c2a40041558002c04cb7864add50abbf465f7ce748189af821e6727e"} Jul 16 11:03:51.901885 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:51.901859 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ss9lt" event={"ID":"daaff1e0-e805-446f-8a08-4baf8c77191d","Type":"ContainerStarted","Data":"8c6fac3ca9292ec60293084daf2485d6355b63b7e278e10636157123522cd969"} Jul 16 11:03:51.904362 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:51.904334 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r76fs" event={"ID":"870684c4-b8b2-48d5-8c2f-97c635fad4df","Type":"ContainerStarted","Data":"7d77342161b065c4b81acf2d518a9c88279be43fa9737a1508bfe302cf56804f"} Jul 16 11:03:51.906558 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:51.906533 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" event={"ID":"450893ed-93e8-4a5d-a9b2-3e71ce176a1e","Type":"ContainerStarted","Data":"e770bb54fc1b1b388f98a3fe835297ba0dcfb70a92bcbcf071d11a24ae87ebb3"} Jul 16 11:03:51.909721 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:51.909698 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsbh6" event={"ID":"d7e80ccb-f7c2-469c-b52c-179371d3da15","Type":"ContainerStarted","Data":"d443290422168fa2adf7d095b3ebdeec5a26aafe00adc1a3fbc9c8605b1b8ded"} Jul 16 11:03:52.190589 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:52.190486 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:52.190757 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:52.190672 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 11:03:52.190821 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:52.190758 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret podName:25a48584-b83e-4cff-a68f-18c383e81fc7 nodeName:}" failed. No retries permitted until 2026-07-16 11:03:54.190735943 +0000 UTC m=+6.024084606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret") pod "global-pull-secret-syncer-fxlr6" (UID: "25a48584-b83e-4cff-a68f-18c383e81fc7") : object "kube-system"/"original-pull-secret" not registered Jul 16 11:03:52.291286 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:52.291246 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w6bg\" (UniqueName: \"kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg\") pod \"network-check-target-c7mlk\" (UID: \"4055addb-97de-4ae7-b8e3-296404d93869\") " pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:03:52.291454 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:52.291321 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs\") pod \"network-metrics-daemon-dvxhb\" (UID: \"2e52bd54-2b39-43e6-8494-8b74da851f0f\") " pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:03:52.291511 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:52.291479 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:03:52.291560 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:52.291542 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs podName:2e52bd54-2b39-43e6-8494-8b74da851f0f nodeName:}" failed. No retries permitted until 2026-07-16 11:03:54.291522816 +0000 UTC m=+6.124871492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs") pod "network-metrics-daemon-dvxhb" (UID: "2e52bd54-2b39-43e6-8494-8b74da851f0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:03:52.291775 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:52.291753 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 11:03:52.291838 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:52.291784 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 11:03:52.291838 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:52.291797 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8w6bg for pod openshift-network-diagnostics/network-check-target-c7mlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:03:52.291925 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:52.291850 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg podName:4055addb-97de-4ae7-b8e3-296404d93869 nodeName:}" failed. No retries permitted until 2026-07-16 11:03:54.291833234 +0000 UTC m=+6.125181931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8w6bg" (UniqueName: "kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg") pod "network-check-target-c7mlk" (UID: "4055addb-97de-4ae7-b8e3-296404d93869") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:03:52.635711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:52.635590 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 10:58:50 +0000 UTC" deadline="2027-12-10 19:43:22.672283119 +0000 UTC" Jul 16 11:03:52.635711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:52.635657 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12296h39m30.036630648s" Jul 16 11:03:52.882464 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:52.882431 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:03:52.882645 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:52.882563 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:03:52.882821 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:52.882669 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:03:52.882821 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:52.882761 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:03:53.881009 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:53.880976 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:53.881713 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:53.881101 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:03:54.206532 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:54.205873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:54.206532 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:54.206038 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 11:03:54.206532 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:54.206127 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret podName:25a48584-b83e-4cff-a68f-18c383e81fc7 nodeName:}" failed. No retries permitted until 2026-07-16 11:03:58.206107922 +0000 UTC m=+10.039456585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret") pod "global-pull-secret-syncer-fxlr6" (UID: "25a48584-b83e-4cff-a68f-18c383e81fc7") : object "kube-system"/"original-pull-secret" not registered Jul 16 11:03:54.307940 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:54.306968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w6bg\" (UniqueName: \"kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg\") pod \"network-check-target-c7mlk\" (UID: \"4055addb-97de-4ae7-b8e3-296404d93869\") " pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:03:54.307940 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:54.307030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs\") pod \"network-metrics-daemon-dvxhb\" (UID: \"2e52bd54-2b39-43e6-8494-8b74da851f0f\") " pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:03:54.307940 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:54.307167 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:03:54.307940 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:54.307230 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs podName:2e52bd54-2b39-43e6-8494-8b74da851f0f nodeName:}" failed. No retries permitted until 2026-07-16 11:03:58.307211404 +0000 UTC m=+10.140560067 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs") pod "network-metrics-daemon-dvxhb" (UID: "2e52bd54-2b39-43e6-8494-8b74da851f0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:03:54.307940 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:54.307757 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 11:03:54.307940 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:54.307777 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 11:03:54.307940 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:54.307791 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8w6bg for pod openshift-network-diagnostics/network-check-target-c7mlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:03:54.307940 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:54.307842 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg podName:4055addb-97de-4ae7-b8e3-296404d93869 nodeName:}" failed. No retries permitted until 2026-07-16 11:03:58.307825204 +0000 UTC m=+10.141173866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8w6bg" (UniqueName: "kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg") pod "network-check-target-c7mlk" (UID: "4055addb-97de-4ae7-b8e3-296404d93869") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:03:54.881441 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:54.880756 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:03:54.881441 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:54.880888 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:03:54.881441 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:54.881280 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:03:54.881441 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:54.881388 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:03:55.880197 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:55.880139 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:55.880383 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:55.880339 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:03:56.880977 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:56.880891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:03:56.880977 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:56.880916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:03:56.881446 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:56.881036 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:03:56.881446 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:56.881148 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:03:57.880225 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:57.880179 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:57.880397 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:57.880322 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:03:58.241280 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:58.241147 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:58.241754 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:58.241307 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 11:03:58.241754 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:58.241372 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret podName:25a48584-b83e-4cff-a68f-18c383e81fc7 nodeName:}" failed. No retries permitted until 2026-07-16 11:04:06.241352088 +0000 UTC m=+18.074700754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret") pod "global-pull-secret-syncer-fxlr6" (UID: "25a48584-b83e-4cff-a68f-18c383e81fc7") : object "kube-system"/"original-pull-secret" not registered Jul 16 11:03:58.342436 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:58.342388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs\") pod \"network-metrics-daemon-dvxhb\" (UID: \"2e52bd54-2b39-43e6-8494-8b74da851f0f\") " pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:03:58.343174 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:58.342490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w6bg\" (UniqueName: \"kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg\") pod \"network-check-target-c7mlk\" (UID: \"4055addb-97de-4ae7-b8e3-296404d93869\") " pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:03:58.343174 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:58.342676 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 11:03:58.343174 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:58.342697 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 11:03:58.343174 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:58.342701 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:03:58.343174 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:58.342786 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs podName:2e52bd54-2b39-43e6-8494-8b74da851f0f nodeName:}" failed. No retries permitted until 2026-07-16 11:04:06.342763289 +0000 UTC m=+18.176111972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs") pod "network-metrics-daemon-dvxhb" (UID: "2e52bd54-2b39-43e6-8494-8b74da851f0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:03:58.343174 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:58.342710 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8w6bg for pod openshift-network-diagnostics/network-check-target-c7mlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:03:58.343174 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:58.342888 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg podName:4055addb-97de-4ae7-b8e3-296404d93869 nodeName:}" failed. No retries permitted until 2026-07-16 11:04:06.342876289 +0000 UTC m=+18.176224949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8w6bg" (UniqueName: "kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg") pod "network-check-target-c7mlk" (UID: "4055addb-97de-4ae7-b8e3-296404d93869") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:03:58.881451 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:58.881410 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:03:58.881648 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:58.881526 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:03:58.882090 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:58.881940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:03:58.882090 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:58.882048 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:03:59.880218 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:03:59.880016 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:03:59.880218 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:03:59.880144 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:00.881117 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:00.881079 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:00.881535 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:00.881080 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:00.881535 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:00.881232 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:00.881535 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:00.881289 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:01.881073 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:01.881028 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:01.881229 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:01.881176 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:02.880455 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:02.880423 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:02.880650 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:02.880546 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:02.880650 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:02.880604 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:02.880770 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:02.880732 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:03.880249 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:03.880214 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:03.880764 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:03.880348 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:04.880551 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:04.880512 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:04.880551 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:04.880538 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:04.881072 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:04.880653 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:04.881072 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:04.880778 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:05.880256 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:05.880217 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:05.880457 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:05.880342 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:06.305668 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:06.305635 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:06.306108 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:06.305808 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 11:04:06.306108 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:06.305887 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret podName:25a48584-b83e-4cff-a68f-18c383e81fc7 nodeName:}" failed. No retries permitted until 2026-07-16 11:04:22.305866533 +0000 UTC m=+34.139215199 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret") pod "global-pull-secret-syncer-fxlr6" (UID: "25a48584-b83e-4cff-a68f-18c383e81fc7") : object "kube-system"/"original-pull-secret" not registered Jul 16 11:04:06.406649 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:06.406552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w6bg\" (UniqueName: \"kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg\") pod \"network-check-target-c7mlk\" (UID: \"4055addb-97de-4ae7-b8e3-296404d93869\") " pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:06.406649 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:06.406632 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs\") pod \"network-metrics-daemon-dvxhb\" (UID: \"2e52bd54-2b39-43e6-8494-8b74da851f0f\") " pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:06.406846 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:06.406728 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 11:04:06.406846 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:06.406751 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:04:06.406846 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:06.406759 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 11:04:06.406846 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:06.406770 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8w6bg for pod openshift-network-diagnostics/network-check-target-c7mlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:04:06.406846 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:06.406815 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs podName:2e52bd54-2b39-43e6-8494-8b74da851f0f nodeName:}" failed. No retries permitted until 2026-07-16 11:04:22.406796304 +0000 UTC m=+34.240144966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs") pod "network-metrics-daemon-dvxhb" (UID: "2e52bd54-2b39-43e6-8494-8b74da851f0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:04:06.406846 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:06.406833 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg podName:4055addb-97de-4ae7-b8e3-296404d93869 nodeName:}" failed. No retries permitted until 2026-07-16 11:04:22.406823532 +0000 UTC m=+34.240172198 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8w6bg" (UniqueName: "kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg") pod "network-check-target-c7mlk" (UID: "4055addb-97de-4ae7-b8e3-296404d93869") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:04:06.880634 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:06.880540 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:06.880634 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:06.880610 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:06.880842 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:06.880706 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:06.880882 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:06.880847 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:07.880397 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:07.880360 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:07.880847 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:07.880471 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:08.881387 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:08.881344 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:08.881851 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:08.881480 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:08.881851 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:08.881550 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:08.881851 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:08.881703 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:09.880667 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:09.880627 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:09.880858 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:09.880769 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:10.880536 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:10.880508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:10.880919 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:10.880633 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:10.880919 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:10.880672 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:10.880919 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:10.880777 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:10.958624 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:10.958498 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" event={"ID":"e3f21bc6-71e7-4101-8f32-f2c0b0ccaca0","Type":"ContainerStarted","Data":"cab5858a700820b0b7ebb4f0c8c79bf7508e921e663fef65f0d896e6e1c43f78"} Jul 16 11:04:10.961529 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:10.961505 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-15.ec2.internal" event={"ID":"b2e3677082925fab7d36c1bc7a29637d","Type":"ContainerStarted","Data":"569bb0da1e0ad268279ad5ce204b9e9e647201b6b264e46c5382095188e6fb05"} Jul 16 11:04:10.997084 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:10.997036 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-15.ec2.internal" podStartSLOduration=20.997020915 podStartE2EDuration="20.997020915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:10.996757237 +0000 UTC m=+22.830105920" watchObservedRunningTime="2026-07-16 11:04:10.997020915 +0000 UTC m=+22.830369609" Jul 16 11:04:10.997188 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:10.997109 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xnd7p" podStartSLOduration=3.119340597 podStartE2EDuration="22.997103311s" totalImagesPullingTime="19.877762714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:10.981716964 +0000 UTC m=+22.815065647" watchObservedRunningTime="2026-07-16 11:04:10.997103311 +0000 UTC m=+22.830451994" Jul 16 11:04:11.880877 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.880665 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:11.881801 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:11.880971 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:11.965869 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.965824 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-95rzf" event={"ID":"61cc2f50-0bdd-400e-92da-09235b2d344b","Type":"ContainerStarted","Data":"3c2c6db40ff15fc652a27407952b4c051b36b8b1b5f0cc71572a738b8d99e4e2"} Jul 16 11:04:11.967467 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.967432 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mn5mk" event={"ID":"2e5ba1b3-7981-43d4-b14a-9ace2f2e1128","Type":"ContainerStarted","Data":"c8e587e7bbf21b295c711f3af8c140d88b61b67727dab91afbdfef582c91e8f4"} Jul 16 11:04:11.969024 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.968999 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ss9lt" event={"ID":"daaff1e0-e805-446f-8a08-4baf8c77191d","Type":"ContainerStarted","Data":"f063a09fc8553281b9fb69ee2fda08c554fbb2dac714ac3ce9592c5228210b14"} Jul 16 11:04:11.970681 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.970657 2580 generic.go:356] "Generic (PLEG): container finished" podID="85ac7ba77ee7d6a135faaf9d8c1bdae9" containerID="3a8bc70f2e58a7037e2501d4418e8f15a1c49a15fdad4b1d1162acb558ddee26" exitCode=0 Jul 16 11:04:11.970808 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.970736 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal" event={"ID":"85ac7ba77ee7d6a135faaf9d8c1bdae9","Type":"ContainerDied","Data":"3a8bc70f2e58a7037e2501d4418e8f15a1c49a15fdad4b1d1162acb558ddee26"} Jul 16 11:04:11.972461 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.972419 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r76fs" event={"ID":"870684c4-b8b2-48d5-8c2f-97c635fad4df","Type":"ContainerStarted","Data":"db4c01ead0f55b878c6efa819f7cba84a879b968f1ab04c409ce3e9b5f82b19e"} Jul 16 11:04:11.974134 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.974110 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" event={"ID":"450893ed-93e8-4a5d-a9b2-3e71ce176a1e","Type":"ContainerStarted","Data":"09bd754b30ca1707a5bd1777214e1d1ec57eb6b2d81b870c6a0f4b3b5cf30bbf"} Jul 16 11:04:11.975638 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.975614 2580 generic.go:356] "Generic (PLEG): container finished" podID="d7e80ccb-f7c2-469c-b52c-179371d3da15" containerID="5a5a58af1a4d18d498b5f9b8671cd6091bd88654f895392e5adf7a1ba78c512e" exitCode=0 Jul 16 11:04:11.975749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.975646 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsbh6" event={"ID":"d7e80ccb-f7c2-469c-b52c-179371d3da15","Type":"ContainerDied","Data":"5a5a58af1a4d18d498b5f9b8671cd6091bd88654f895392e5adf7a1ba78c512e"} Jul 16 11:04:11.978701 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.978680 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:04:11.979393 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.978996 2580 generic.go:356] "Generic (PLEG): container finished" podID="04bb84b9-bd39-419b-947a-2ab4c9d0476d" containerID="755210cfc1637bcd09f655b1a6da521430a13378d3691f474268794f62031f43" exitCode=1 Jul 16 11:04:11.979393 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.979073 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" event={"ID":"04bb84b9-bd39-419b-947a-2ab4c9d0476d","Type":"ContainerStarted","Data":"c0c484da758bd0f2809d6ffa1f79e1d96067395b2636bd0d94c2b13b896bf3b0"} Jul 16 11:04:11.979393 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.979146 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" event={"ID":"04bb84b9-bd39-419b-947a-2ab4c9d0476d","Type":"ContainerStarted","Data":"746ea916a898f6d31280da9fc5cf540c80668477d841e463c58ba4e73df2ac8b"} Jul 16 11:04:11.979393 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.979197 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" event={"ID":"04bb84b9-bd39-419b-947a-2ab4c9d0476d","Type":"ContainerStarted","Data":"f2c28ac1a13d503a254040f57a65611a94f9347cfe960f530306016bfe1d287e"} Jul 16 11:04:11.979393 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.979245 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" event={"ID":"04bb84b9-bd39-419b-947a-2ab4c9d0476d","Type":"ContainerStarted","Data":"7191e8bb4cacb878c07decdb82af1d2977b67c78c109c3a890cddfb4a53bfaed"} Jul 16 11:04:11.979393 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.979293 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" event={"ID":"04bb84b9-bd39-419b-947a-2ab4c9d0476d","Type":"ContainerStarted","Data":"3beeb9c30b52d785b177eec65b58c280d382b0f3035b225e4f2f5144d42845fa"} Jul 16 11:04:11.979393 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.979346 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" event={"ID":"04bb84b9-bd39-419b-947a-2ab4c9d0476d","Type":"ContainerDied","Data":"755210cfc1637bcd09f655b1a6da521430a13378d3691f474268794f62031f43"} Jul 16 11:04:11.985791 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:11.985752 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-95rzf" podStartSLOduration=2.588751981 podStartE2EDuration="22.985739788s" totalImagesPullingTime="20.396987807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:11.985116479 +0000 UTC m=+23.818465161" watchObservedRunningTime="2026-07-16 11:04:11.985739788 +0000 UTC m=+23.819088470" Jul 16 11:04:12.000998 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:12.000940 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r76fs" podStartSLOduration=3.238266561 podStartE2EDuration="23.000921446s" totalImagesPullingTime="19.762654885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:12.000444976 +0000 UTC m=+23.833793686" watchObservedRunningTime="2026-07-16 11:04:12.000921446 +0000 UTC m=+23.834270128" Jul 16 11:04:12.025474 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:12.024979 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ss9lt" podStartSLOduration=7.838095007 podStartE2EDuration="23.02495936s" totalImagesPullingTime="15.186864353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:12.024781923 +0000 UTC m=+23.858130605" watchObservedRunningTime="2026-07-16 11:04:12.02495936 +0000 UTC m=+23.858308043" Jul 16 11:04:12.096722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:12.096653 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mn5mk" podStartSLOduration=4.283533703 podStartE2EDuration="24.096633551s" totalImagesPullingTime="19.813099848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:12.096605138 +0000 UTC m=+23.929953820" watchObservedRunningTime="2026-07-16 11:04:12.096633551 +0000 UTC m=+23.929982227" Jul 16 11:04:12.562443 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:12.562415 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 11:04:12.703731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:12.703599 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T11:04:12.562438403Z","UUID":"3d199220-ff3e-473e-a609-432f7444eb9b","Handler":null,"Name":"","Endpoint":""} Jul 16 11:04:12.705101 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:12.705077 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 11:04:12.705221 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:12.705116 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 11:04:12.880915 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:12.880885 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:12.880915 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:12.880924 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:12.881549 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:12.881018 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:12.881549 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:12.881375 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:12.983998 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:12.983899 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal" event={"ID":"85ac7ba77ee7d6a135faaf9d8c1bdae9","Type":"ContainerStarted","Data":"18b088196f0eeeacc724108208853ca2c011847d0132afd3730ee1fe8b47ea25"} Jul 16 11:04:12.985864 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:12.985838 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" event={"ID":"450893ed-93e8-4a5d-a9b2-3e71ce176a1e","Type":"ContainerStarted","Data":"48217d43ace4ea6f6c786e4f41dd1fe380cde34100c460f6e6d1e7d06dddf793"} Jul 16 11:04:12.987498 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:12.987466 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lwzs8" event={"ID":"eb9a9276-10aa-4cee-844f-46bc4494697b","Type":"ContainerStarted","Data":"f6773f816452c854e28c2dda95bbeb33d984da989ebac11fc0ef832f14a2e018"} Jul 16 11:04:13.001909 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:13.001854 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-15.ec2.internal" podStartSLOduration=23.001836427 podStartE2EDuration="23.001836427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:13.001363657 +0000 UTC m=+24.834712339" watchObservedRunningTime="2026-07-16 11:04:13.001836427 +0000 UTC m=+24.835185110" Jul 16 11:04:13.017722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:13.017673 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lwzs8" podStartSLOduration=4.19356849 podStartE2EDuration="24.017656567s" totalImagesPullingTime="19.824088077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:13.017138938 +0000 UTC m=+24.850487617" watchObservedRunningTime="2026-07-16 11:04:13.017656567 +0000 UTC m=+24.851005254" Jul 16 11:04:13.880679 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:13.880647 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:13.880876 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:13.880774 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:13.996507 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:13.996475 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:04:13.997013 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:13.996886 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" event={"ID":"04bb84b9-bd39-419b-947a-2ab4c9d0476d","Type":"ContainerStarted","Data":"844a35b76073f34e7dc5d1685e48688adc1b7e2d94a8aa3392bb8ecf4a5b3751"} Jul 16 11:04:14.880697 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:14.880668 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:14.880973 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:14.880729 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:14.880973 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:14.880822 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:14.881073 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:14.880976 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:15.001901 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:15.001863 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" event={"ID":"450893ed-93e8-4a5d-a9b2-3e71ce176a1e","Type":"ContainerStarted","Data":"7494b209bf0a06c7e3b8b502471353d535856a9ed9055a01efe7ab66dc9ce469"} Jul 16 11:04:15.027630 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:15.027558 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26bwf" podStartSLOduration=3.7332896079999998 podStartE2EDuration="26.027541452s" totalImagesPullingTime="22.294251844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:03:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:15.027381737 +0000 UTC m=+26.860730446" watchObservedRunningTime="2026-07-16 11:04:15.027541452 +0000 UTC m=+26.860890138" Jul 16 11:04:15.880872 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:15.880833 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:15.881073 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:15.880969 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:16.009132 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:16.008922 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:04:16.010360 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:16.010245 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" event={"ID":"04bb84b9-bd39-419b-947a-2ab4c9d0476d","Type":"ContainerStarted","Data":"1f1899d9325bf7a294f3e467890dcfa4679478e5407c4f52e3a60ec28d631aa0"} Jul 16 11:04:16.010712 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:16.010586 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:04:16.010712 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:16.010659 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:04:16.010712 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:16.010704 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:04:16.010994 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:16.010847 2580 scope.go:117] "RemoveContainer" containerID="755210cfc1637bcd09f655b1a6da521430a13378d3691f474268794f62031f43" Jul 16 11:04:16.036687 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:16.030040 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:04:16.036687 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:16.031740 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:04:16.763741 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:16.763717 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mn5mk" Jul 16 11:04:16.764335 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:16.764315 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mn5mk" Jul 16 11:04:16.880335 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:16.880304 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:16.880497 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:16.880422 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:16.880497 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:16.880472 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:16.880648 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:16.880562 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:17.015425 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:17.015358 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:04:17.016267 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:17.015796 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" event={"ID":"04bb84b9-bd39-419b-947a-2ab4c9d0476d","Type":"ContainerStarted","Data":"7169794b3629a2ace02726d0de3b84c2884aae6df40c825b86e4d71fbe2379cf"} Jul 16 11:04:17.017513 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:17.017489 2580 generic.go:356] "Generic (PLEG): container finished" podID="d7e80ccb-f7c2-469c-b52c-179371d3da15" containerID="7e1691bbf23347bcc7a8185739aa39daa5efcb2c584d6974358360bdd3348ff2" exitCode=0 Jul 16 11:04:17.017629 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:17.017593 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsbh6" event={"ID":"d7e80ccb-f7c2-469c-b52c-179371d3da15","Type":"ContainerDied","Data":"7e1691bbf23347bcc7a8185739aa39daa5efcb2c584d6974358360bdd3348ff2"} Jul 16 11:04:17.017761 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:17.017745 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mn5mk" Jul 16 11:04:17.018622 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:17.018325 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mn5mk" Jul 16 11:04:17.052183 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:17.052144 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" podStartSLOduration=8.163968802 podStartE2EDuration="28.052132216s" totalImagesPullingTime="19.888163414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:17.050749497 +0000 UTC m=+28.884098170" watchObservedRunningTime="2026-07-16 11:04:17.052132216 +0000 UTC m=+28.885480903" Jul 16 11:04:17.543302 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:17.543269 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ss9lt_daaff1e0-e805-446f-8a08-4baf8c77191d/dns-node-resolver/0.log" Jul 16 11:04:17.880739 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:17.880658 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:17.880999 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:17.880771 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:18.526320 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:18.526168 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r76fs_870684c4-b8b2-48d5-8c2f-97c635fad4df/node-ca/0.log" Jul 16 11:04:18.676178 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:18.676147 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fxlr6"] Jul 16 11:04:18.676341 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:18.676275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:18.676411 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:18.676387 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:18.679984 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:18.679949 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c7mlk"] Jul 16 11:04:18.680132 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:18.680070 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:18.680198 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:18.680160 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:18.680546 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:18.680512 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dvxhb"] Jul 16 11:04:18.680702 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:18.680636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:18.680773 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:18.680741 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:19.023824 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:19.023789 2580 generic.go:356] "Generic (PLEG): container finished" podID="d7e80ccb-f7c2-469c-b52c-179371d3da15" containerID="f7f01e70b06b1799dd0d107abfdc6f22a90c0b3a54e99e81e46c9a254c07b250" exitCode=0 Jul 16 11:04:19.023978 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:19.023870 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsbh6" event={"ID":"d7e80ccb-f7c2-469c-b52c-179371d3da15","Type":"ContainerDied","Data":"f7f01e70b06b1799dd0d107abfdc6f22a90c0b3a54e99e81e46c9a254c07b250"} Jul 16 11:04:20.880433 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:20.880395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:20.880945 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:20.880401 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:20.880945 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:20.880508 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:20.880945 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:20.880410 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:20.880945 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:20.880592 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:20.880945 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:20.880680 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:21.031031 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:21.030993 2580 generic.go:356] "Generic (PLEG): container finished" podID="d7e80ccb-f7c2-469c-b52c-179371d3da15" containerID="db6f7af9ebba3a1f650d0e6d4e97b0b1dbd4bef75a8c4a8f731e948f0beeb742" exitCode=0 Jul 16 11:04:21.031194 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:21.031060 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsbh6" event={"ID":"d7e80ccb-f7c2-469c-b52c-179371d3da15","Type":"ContainerDied","Data":"db6f7af9ebba3a1f650d0e6d4e97b0b1dbd4bef75a8c4a8f731e948f0beeb742"} Jul 16 11:04:22.323117 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:22.323080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:22.323532 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:22.323250 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 11:04:22.323532 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:22.323350 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret podName:25a48584-b83e-4cff-a68f-18c383e81fc7 nodeName:}" failed. No retries permitted until 2026-07-16 11:04:54.3233276 +0000 UTC m=+66.156676263 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret") pod "global-pull-secret-syncer-fxlr6" (UID: "25a48584-b83e-4cff-a68f-18c383e81fc7") : object "kube-system"/"original-pull-secret" not registered Jul 16 11:04:22.423948 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:22.423909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w6bg\" (UniqueName: \"kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg\") pod \"network-check-target-c7mlk\" (UID: \"4055addb-97de-4ae7-b8e3-296404d93869\") " pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:22.424100 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:22.423980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs\") pod \"network-metrics-daemon-dvxhb\" (UID: \"2e52bd54-2b39-43e6-8494-8b74da851f0f\") " pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:22.424159 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:22.424100 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:04:22.424159 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:22.424113 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 11:04:22.424159 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:22.424155 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs podName:2e52bd54-2b39-43e6-8494-8b74da851f0f nodeName:}" failed. No retries permitted until 2026-07-16 11:04:54.424140319 +0000 UTC m=+66.257488979 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs") pod "network-metrics-daemon-dvxhb" (UID: "2e52bd54-2b39-43e6-8494-8b74da851f0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:04:22.424301 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:22.424202 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 11:04:22.424301 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:22.424220 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8w6bg for pod openshift-network-diagnostics/network-check-target-c7mlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:04:22.424301 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:22.424272 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg podName:4055addb-97de-4ae7-b8e3-296404d93869 nodeName:}" failed. No retries permitted until 2026-07-16 11:04:54.424259969 +0000 UTC m=+66.257608633 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8w6bg" (UniqueName: "kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg") pod "network-check-target-c7mlk" (UID: "4055addb-97de-4ae7-b8e3-296404d93869") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:04:22.881112 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:22.881068 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:22.881301 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:22.881068 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:22.881301 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:22.881192 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:22.881301 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:22.881223 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:22.881418 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:22.881339 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:22.881449 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:22.881413 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:24.880460 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:24.880418 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:24.880955 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:24.880541 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:24.880955 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:24.880552 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:24.880955 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:24.880598 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:24.880955 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:24.880696 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:24.880955 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:24.880794 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:25.043987 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:25.043821 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsbh6" event={"ID":"d7e80ccb-f7c2-469c-b52c-179371d3da15","Type":"ContainerStarted","Data":"3e361afe3f7cf42e33cfc6be5697044a974bb490a7f071a10446f5e1db415eb0"} Jul 16 11:04:26.048486 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:26.048453 2580 generic.go:356] "Generic (PLEG): container finished" podID="d7e80ccb-f7c2-469c-b52c-179371d3da15" containerID="3e361afe3f7cf42e33cfc6be5697044a974bb490a7f071a10446f5e1db415eb0" exitCode=0 Jul 16 11:04:26.048963 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:26.048520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsbh6" event={"ID":"d7e80ccb-f7c2-469c-b52c-179371d3da15","Type":"ContainerDied","Data":"3e361afe3f7cf42e33cfc6be5697044a974bb490a7f071a10446f5e1db415eb0"} Jul 16 11:04:26.880554 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:26.880516 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:26.880761 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:26.880516 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:26.880761 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:26.880646 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:26.880761 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:26.880659 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:26.880761 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:26.880715 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:26.880919 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:26.880802 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:27.053494 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:27.053462 2580 generic.go:356] "Generic (PLEG): container finished" podID="d7e80ccb-f7c2-469c-b52c-179371d3da15" containerID="764ae43dd0b568b6a996647859fc012aa72fb443ef52c7babf8f77926018bc9b" exitCode=0 Jul 16 11:04:27.053881 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:27.053518 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsbh6" event={"ID":"d7e80ccb-f7c2-469c-b52c-179371d3da15","Type":"ContainerDied","Data":"764ae43dd0b568b6a996647859fc012aa72fb443ef52c7babf8f77926018bc9b"} Jul 16 11:04:28.058992 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:28.058958 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsbh6" event={"ID":"d7e80ccb-f7c2-469c-b52c-179371d3da15","Type":"ContainerStarted","Data":"260b645fb344c2b2c5dffcfd8002aa8e3d2e197003b2535b3b6a0a83801e4123"} Jul 16 11:04:28.083055 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:28.083003 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dsbh6" podStartSLOduration=9.118255865 podStartE2EDuration="39.082988694s" totalImagesPullingTime="29.964732829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:03:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:28.082831476 +0000 UTC m=+39.916180172" watchObservedRunningTime="2026-07-16 11:04:28.082988694 +0000 UTC m=+39.916337376" Jul 16 11:04:28.881850 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:28.881822 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:28.882015 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:28.881913 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:28.882015 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:28.881952 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:28.882096 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:28.882076 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:28.882130 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:28.882112 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:28.882200 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:28.882184 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:30.880180 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:30.880140 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:30.880769 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:30.880201 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:30.880769 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:30.880290 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:30.880769 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:30.880294 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:30.880769 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:30.880398 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:30.880769 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:30.880495 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:32.881123 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:32.881090 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:32.881507 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:32.881090 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:32.881507 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:32.881091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:32.881507 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:32.881271 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:32.881507 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:32.881358 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:32.881507 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:32.881418 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:34.880281 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:34.880237 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:34.880896 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:34.880244 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:34.880896 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:34.880356 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:34.880896 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:34.880462 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:34.880896 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:34.880244 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:34.880896 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:34.880613 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:36.881033 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:36.881001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:36.881509 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:36.881007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:36.881509 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:36.881117 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:36.881509 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:36.881009 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:36.881509 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:36.881180 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:36.881509 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:36.881242 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:38.881534 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:38.881493 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:38.882016 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:38.881623 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7mlk" podUID="4055addb-97de-4ae7-b8e3-296404d93869" Jul 16 11:04:38.882016 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:38.881715 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:38.882316 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:38.882282 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:38.882743 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:38.882548 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dvxhb" podUID="2e52bd54-2b39-43e6-8494-8b74da851f0f" Jul 16 11:04:38.884627 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:38.884313 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxlr6" podUID="25a48584-b83e-4cff-a68f-18c383e81fc7" Jul 16 11:04:40.013586 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.013557 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-15.ec2.internal" event="NodeReady" Jul 16 11:04:40.013939 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.013704 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 11:04:40.083670 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.083638 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nxfs5"] Jul 16 11:04:40.096981 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.096950 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.099394 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.099375 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 11:04:40.099974 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.099953 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 11:04:40.100200 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.100185 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 11:04:40.100257 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.100185 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lhr85\"" Jul 16 11:04:40.100496 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.100482 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 11:04:40.101952 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.101930 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nxfs5"] Jul 16 11:04:40.106532 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.106511 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rb76j"] Jul 16 11:04:40.118980 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.118957 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ddhln"] Jul 16 11:04:40.119106 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.119089 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rb76j" Jul 16 11:04:40.121610 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.121585 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 11:04:40.121947 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.121929 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 11:04:40.122045 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.121948 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 11:04:40.122045 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.121948 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-45jx4\"" Jul 16 11:04:40.137256 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.137227 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rb76j"] Jul 16 11:04:40.137384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.137271 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ddhln"] Jul 16 11:04:40.137384 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.137340 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:40.140918 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.140896 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 11:04:40.141013 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.140897 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kmdfj\"" Jul 16 11:04:40.141013 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.140897 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 11:04:40.152869 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.152841 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6c40a24-95a6-4c49-b0b2-9140c8e05b4b-cert\") pod \"ingress-canary-rb76j\" (UID: \"d6c40a24-95a6-4c49-b0b2-9140c8e05b4b\") " pod="openshift-ingress-canary/ingress-canary-rb76j" Jul 16 11:04:40.152996 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.152884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00196644-3e96-4e0a-9002-ac8d51e53cc1-crio-socket\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.152996 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.152948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34c50701-dc77-4154-895c-cf88dfd93f5e-metrics-tls\") pod \"dns-default-ddhln\" (UID: \"34c50701-dc77-4154-895c-cf88dfd93f5e\") " pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:40.152996 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.152977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34c50701-dc77-4154-895c-cf88dfd93f5e-tmp-dir\") pod \"dns-default-ddhln\" (UID: \"34c50701-dc77-4154-895c-cf88dfd93f5e\") " pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:40.153160 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.153000 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skq24\" (UniqueName: \"kubernetes.io/projected/34c50701-dc77-4154-895c-cf88dfd93f5e-kube-api-access-skq24\") pod \"dns-default-ddhln\" (UID: \"34c50701-dc77-4154-895c-cf88dfd93f5e\") " pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:40.153160 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.153028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00196644-3e96-4e0a-9002-ac8d51e53cc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.153160 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.153052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq94k\" (UniqueName: \"kubernetes.io/projected/00196644-3e96-4e0a-9002-ac8d51e53cc1-kube-api-access-xq94k\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.153160 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.153080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34c50701-dc77-4154-895c-cf88dfd93f5e-config-volume\") pod \"dns-default-ddhln\" (UID: \"34c50701-dc77-4154-895c-cf88dfd93f5e\") " pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:40.153160 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.153109 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txdqq\" (UniqueName: \"kubernetes.io/projected/d6c40a24-95a6-4c49-b0b2-9140c8e05b4b-kube-api-access-txdqq\") pod \"ingress-canary-rb76j\" (UID: \"d6c40a24-95a6-4c49-b0b2-9140c8e05b4b\") " pod="openshift-ingress-canary/ingress-canary-rb76j" Jul 16 11:04:40.153160 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.153155 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/00196644-3e96-4e0a-9002-ac8d51e53cc1-data-volume\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.153360 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.153189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00196644-3e96-4e0a-9002-ac8d51e53cc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.254355 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.254325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34c50701-dc77-4154-895c-cf88dfd93f5e-metrics-tls\") pod \"dns-default-ddhln\" (UID: \"34c50701-dc77-4154-895c-cf88dfd93f5e\") " pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:40.254355 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.254366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34c50701-dc77-4154-895c-cf88dfd93f5e-tmp-dir\") pod \"dns-default-ddhln\" (UID: \"34c50701-dc77-4154-895c-cf88dfd93f5e\") " pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:40.254638 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.254392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skq24\" (UniqueName: \"kubernetes.io/projected/34c50701-dc77-4154-895c-cf88dfd93f5e-kube-api-access-skq24\") pod \"dns-default-ddhln\" (UID: \"34c50701-dc77-4154-895c-cf88dfd93f5e\") " pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:40.254638 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.254420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00196644-3e96-4e0a-9002-ac8d51e53cc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.254638 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.254443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq94k\" (UniqueName: \"kubernetes.io/projected/00196644-3e96-4e0a-9002-ac8d51e53cc1-kube-api-access-xq94k\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.254638 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.254471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34c50701-dc77-4154-895c-cf88dfd93f5e-config-volume\") pod \"dns-default-ddhln\" (UID: \"34c50701-dc77-4154-895c-cf88dfd93f5e\") " pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:40.254638 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.254500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txdqq\" (UniqueName: \"kubernetes.io/projected/d6c40a24-95a6-4c49-b0b2-9140c8e05b4b-kube-api-access-txdqq\") pod \"ingress-canary-rb76j\" (UID: \"d6c40a24-95a6-4c49-b0b2-9140c8e05b4b\") " pod="openshift-ingress-canary/ingress-canary-rb76j" Jul 16 11:04:40.254638 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.254532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/00196644-3e96-4e0a-9002-ac8d51e53cc1-data-volume\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.254638 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.254553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00196644-3e96-4e0a-9002-ac8d51e53cc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.254638 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.254628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6c40a24-95a6-4c49-b0b2-9140c8e05b4b-cert\") pod \"ingress-canary-rb76j\" (UID: \"d6c40a24-95a6-4c49-b0b2-9140c8e05b4b\") " pod="openshift-ingress-canary/ingress-canary-rb76j" Jul 16 11:04:40.254959 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.254663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00196644-3e96-4e0a-9002-ac8d51e53cc1-crio-socket\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.254959 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.254826 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34c50701-dc77-4154-895c-cf88dfd93f5e-tmp-dir\") pod \"dns-default-ddhln\" (UID: \"34c50701-dc77-4154-895c-cf88dfd93f5e\") " pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:40.254959 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.254900 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00196644-3e96-4e0a-9002-ac8d51e53cc1-crio-socket\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.254959 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.254913 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/00196644-3e96-4e0a-9002-ac8d51e53cc1-data-volume\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.255108 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.255048 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34c50701-dc77-4154-895c-cf88dfd93f5e-config-volume\") pod \"dns-default-ddhln\" (UID: \"34c50701-dc77-4154-895c-cf88dfd93f5e\") " pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:40.255814 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.255790 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00196644-3e96-4e0a-9002-ac8d51e53cc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.260160 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.260135 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00196644-3e96-4e0a-9002-ac8d51e53cc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.260160 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.260147 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34c50701-dc77-4154-895c-cf88dfd93f5e-metrics-tls\") pod \"dns-default-ddhln\" (UID: \"34c50701-dc77-4154-895c-cf88dfd93f5e\") " pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:40.260759 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.260742 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6c40a24-95a6-4c49-b0b2-9140c8e05b4b-cert\") pod \"ingress-canary-rb76j\" (UID: \"d6c40a24-95a6-4c49-b0b2-9140c8e05b4b\") " pod="openshift-ingress-canary/ingress-canary-rb76j" Jul 16 11:04:40.268694 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.268633 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skq24\" (UniqueName: \"kubernetes.io/projected/34c50701-dc77-4154-895c-cf88dfd93f5e-kube-api-access-skq24\") pod \"dns-default-ddhln\" (UID: \"34c50701-dc77-4154-895c-cf88dfd93f5e\") " pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:40.269122 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.269105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txdqq\" (UniqueName: \"kubernetes.io/projected/d6c40a24-95a6-4c49-b0b2-9140c8e05b4b-kube-api-access-txdqq\") pod \"ingress-canary-rb76j\" (UID: \"d6c40a24-95a6-4c49-b0b2-9140c8e05b4b\") " pod="openshift-ingress-canary/ingress-canary-rb76j" Jul 16 11:04:40.269315 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.269293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq94k\" (UniqueName: \"kubernetes.io/projected/00196644-3e96-4e0a-9002-ac8d51e53cc1-kube-api-access-xq94k\") pod \"insights-runtime-extractor-nxfs5\" (UID: \"00196644-3e96-4e0a-9002-ac8d51e53cc1\") " pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.419449 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.419412 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nxfs5" Jul 16 11:04:40.428526 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.428489 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rb76j" Jul 16 11:04:40.445719 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.445691 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:40.640563 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.640538 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nxfs5"] Jul 16 11:04:40.658615 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.658568 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rb76j"] Jul 16 11:04:40.664156 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:04:40.664130 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c40a24_95a6_4c49_b0b2_9140c8e05b4b.slice/crio-dd0fa3c2424339fdbb34604081b6c5b1f3fe76dad5219a66422d68736dafc75a WatchSource:0}: Error finding container dd0fa3c2424339fdbb34604081b6c5b1f3fe76dad5219a66422d68736dafc75a: Status 404 returned error can't find the container with id dd0fa3c2424339fdbb34604081b6c5b1f3fe76dad5219a66422d68736dafc75a Jul 16 11:04:40.682150 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.682124 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ddhln"] Jul 16 11:04:40.685265 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:04:40.685241 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c50701_dc77_4154_895c_cf88dfd93f5e.slice/crio-09f85c5fdb28b032ad05d809b3cdac84a692b5b7afd5c82839ed18e5c64b6d41 WatchSource:0}: Error finding container 09f85c5fdb28b032ad05d809b3cdac84a692b5b7afd5c82839ed18e5c64b6d41: Status 404 returned error can't find the container with id 09f85c5fdb28b032ad05d809b3cdac84a692b5b7afd5c82839ed18e5c64b6d41 Jul 16 11:04:40.880412 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.880325 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:40.880412 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.880363 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:40.880646 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.880325 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:40.883126 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.883092 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 11:04:40.883260 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.883091 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 11:04:40.883260 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.883098 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rdfxb\"" Jul 16 11:04:40.883260 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.883240 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 11:04:40.883424 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.883317 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 11:04:40.883424 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:40.883372 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-49sx5\"" Jul 16 11:04:41.087511 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.087468 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rb76j" event={"ID":"d6c40a24-95a6-4c49-b0b2-9140c8e05b4b","Type":"ContainerStarted","Data":"dd0fa3c2424339fdbb34604081b6c5b1f3fe76dad5219a66422d68736dafc75a"} Jul 16 11:04:41.088444 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.088424 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nxfs5" event={"ID":"00196644-3e96-4e0a-9002-ac8d51e53cc1","Type":"ContainerStarted","Data":"dd3968d537fa50aa5e47b674547ec8389e4c2b9e832466e3c849502c4bb9225a"} Jul 16 11:04:41.089293 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.089275 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ddhln" event={"ID":"34c50701-dc77-4154-895c-cf88dfd93f5e","Type":"ContainerStarted","Data":"09f85c5fdb28b032ad05d809b3cdac84a692b5b7afd5c82839ed18e5c64b6d41"} Jul 16 11:04:41.311705 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.311663 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c76d5874-lsbs4"] Jul 16 11:04:41.331343 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.330555 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c76d5874-lsbs4"] Jul 16 11:04:41.331343 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.330642 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.333813 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.333784 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 11:04:41.335193 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.334614 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 11:04:41.335193 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.334700 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 11:04:41.335193 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.334701 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 11:04:41.335193 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.335017 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 11:04:41.335475 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.335244 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 11:04:41.335475 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.335352 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bst9c\"" Jul 16 11:04:41.335475 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.335463 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 11:04:41.339847 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.339759 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 11:04:41.365752 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.365717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr8p2\" (UniqueName: \"kubernetes.io/projected/b3f323d8-8779-4dcf-9e15-e1a55b70317d-kube-api-access-fr8p2\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.365752 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.365766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-trusted-ca-bundle\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.366008 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.365799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-oauth-config\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.366008 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.365887 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-serving-cert\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.366008 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.365919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-config\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.366008 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.365942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-service-ca\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.366163 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.366004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-oauth-serving-cert\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.467152 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.467112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr8p2\" (UniqueName: \"kubernetes.io/projected/b3f323d8-8779-4dcf-9e15-e1a55b70317d-kube-api-access-fr8p2\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.467152 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.467157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-trusted-ca-bundle\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.467490 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.467184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-oauth-config\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.467490 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.467211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-serving-cert\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.467490 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.467235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-config\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.467490 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.467257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-service-ca\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.467490 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.467277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-oauth-serving-cert\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.468128 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.468100 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-oauth-serving-cert\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.468341 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.468315 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-trusted-ca-bundle\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.468412 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.468355 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-config\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.468583 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.468544 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-service-ca\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.471367 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.471339 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-oauth-config\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.471809 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.471772 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-serving-cert\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.481720 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.481690 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr8p2\" (UniqueName: \"kubernetes.io/projected/b3f323d8-8779-4dcf-9e15-e1a55b70317d-kube-api-access-fr8p2\") pod \"console-7c76d5874-lsbs4\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.643450 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.643364 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:41.854517 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:41.854471 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c76d5874-lsbs4"] Jul 16 11:04:41.858825 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:04:41.858781 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f323d8_8779_4dcf_9e15_e1a55b70317d.slice/crio-a483e93129d82ca825c78b7cfde2da94f3e3648e8108f0f0e4a1cc9771db7f9a WatchSource:0}: Error finding container a483e93129d82ca825c78b7cfde2da94f3e3648e8108f0f0e4a1cc9771db7f9a: Status 404 returned error can't find the container with id a483e93129d82ca825c78b7cfde2da94f3e3648e8108f0f0e4a1cc9771db7f9a Jul 16 11:04:42.095722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:42.095683 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c76d5874-lsbs4" event={"ID":"b3f323d8-8779-4dcf-9e15-e1a55b70317d","Type":"ContainerStarted","Data":"a483e93129d82ca825c78b7cfde2da94f3e3648e8108f0f0e4a1cc9771db7f9a"} Jul 16 11:04:47.322053 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:47.322016 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jx876"] Jul 16 11:04:47.323982 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:47.323951 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jx876" Jul 16 11:04:47.327221 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:47.327198 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ljjs8\"" Jul 16 11:04:47.327397 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:47.327373 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 16 11:04:47.334112 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:47.334046 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jx876"] Jul 16 11:04:47.415863 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:47.415777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/90ade4f6-eb51-4a6e-b54b-f30b10701c81-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-jx876\" (UID: \"90ade4f6-eb51-4a6e-b54b-f30b10701c81\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jx876" Jul 16 11:04:47.516816 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:47.516772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/90ade4f6-eb51-4a6e-b54b-f30b10701c81-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-jx876\" (UID: \"90ade4f6-eb51-4a6e-b54b-f30b10701c81\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jx876" Jul 16 11:04:47.519588 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:47.519541 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/90ade4f6-eb51-4a6e-b54b-f30b10701c81-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-jx876\" (UID: \"90ade4f6-eb51-4a6e-b54b-f30b10701c81\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jx876" Jul 16 11:04:47.635789 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:47.635747 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jx876" Jul 16 11:04:48.030865 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:48.030837 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66mkq" Jul 16 11:04:48.718174 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:48.718072 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jx876"] Jul 16 11:04:49.116512 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:49.116480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c76d5874-lsbs4" event={"ID":"b3f323d8-8779-4dcf-9e15-e1a55b70317d","Type":"ContainerStarted","Data":"c476993975558f84715d4a1fa2e33311e3e6799dd418e32e2b5d213145016882"} Jul 16 11:04:49.118280 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:49.118255 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ddhln" event={"ID":"34c50701-dc77-4154-895c-cf88dfd93f5e","Type":"ContainerStarted","Data":"0a5140fb1607d98a671aed5d96fc51890517d749c835e5efcb735a88ae457de9"} Jul 16 11:04:49.118404 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:49.118301 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ddhln" event={"ID":"34c50701-dc77-4154-895c-cf88dfd93f5e","Type":"ContainerStarted","Data":"012617723d46cfa2c86909cb4017397e1aba87f94f1194235e9140fb3881a5e6"} Jul 16 11:04:49.118404 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:49.118322 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:49.119455 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:49.119432 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jx876" event={"ID":"90ade4f6-eb51-4a6e-b54b-f30b10701c81","Type":"ContainerStarted","Data":"82fe6bac8722557a05e6a767792132be048de720c93b08315ec86931cb34f086"} Jul 16 11:04:49.120765 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:49.120738 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rb76j" event={"ID":"d6c40a24-95a6-4c49-b0b2-9140c8e05b4b","Type":"ContainerStarted","Data":"af4fae6e6b11bc39dd6c4d6e02aab506b5bb7683ea903b1b0027f4d6629fa945"} Jul 16 11:04:49.122115 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:49.122096 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nxfs5" event={"ID":"00196644-3e96-4e0a-9002-ac8d51e53cc1","Type":"ContainerStarted","Data":"fb730969cc7c9ff5b89c1e0afaea246b2b354e9c3e33a3dbb5a41c82cdfb0170"} Jul 16 11:04:49.136534 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:49.136489 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c76d5874-lsbs4" podStartSLOduration=1.431107596 podStartE2EDuration="8.136474983s" totalImagesPullingTime="6.705367387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:49.13593774 +0000 UTC m=+60.969286422" watchObservedRunningTime="2026-07-16 11:04:49.136474983 +0000 UTC m=+60.969823655" Jul 16 11:04:49.151629 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:49.151557 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rb76j" podStartSLOduration=1.605476524 podStartE2EDuration="9.151544079s" totalImagesPullingTime="7.546067555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:49.150775006 +0000 UTC m=+60.984123688" watchObservedRunningTime="2026-07-16 11:04:49.151544079 +0000 UTC m=+60.984892761" Jul 16 11:04:49.169957 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:49.169901 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ddhln" podStartSLOduration=2.699965349 podStartE2EDuration="9.169881372s" totalImagesPullingTime="6.469916023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:49.169072839 +0000 UTC m=+61.002421532" watchObservedRunningTime="2026-07-16 11:04:49.169881372 +0000 UTC m=+61.003230053" Jul 16 11:04:50.128782 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:50.128731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jx876" event={"ID":"90ade4f6-eb51-4a6e-b54b-f30b10701c81","Type":"ContainerStarted","Data":"25c8c8fc43e115df7490f54bd6efb1db16db77dde7d9b3fe170e432a35721334"} Jul 16 11:04:50.129269 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:50.129088 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jx876" Jul 16 11:04:50.134539 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:50.134516 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jx876" Jul 16 11:04:50.148712 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:50.148361 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jx876" podStartSLOduration=1.933573982 podStartE2EDuration="3.148344438s" totalImagesPullingTime="1.214770456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:50.147711851 +0000 UTC m=+61.981060534" watchObservedRunningTime="2026-07-16 11:04:50.148344438 +0000 UTC m=+61.981693130" Jul 16 11:04:51.644448 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:51.644402 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:51.644888 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:51.644518 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:04:51.646059 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:51.646034 2580 patch_prober.go:28] interesting pod/console-7c76d5874-lsbs4 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body= Jul 16 11:04:51.646169 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:51.646082 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7c76d5874-lsbs4" podUID="b3f323d8-8779-4dcf-9e15-e1a55b70317d" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" Jul 16 11:04:53.140016 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:53.139986 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nxfs5" event={"ID":"00196644-3e96-4e0a-9002-ac8d51e53cc1","Type":"ContainerStarted","Data":"beab3c7ea677bc0b88192a0aa1457a193586a610784cc4d15ebb968549dd1dff"} Jul 16 11:04:54.371733 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.371695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:54.374477 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.374451 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 11:04:54.385015 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.384988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25a48584-b83e-4cff-a68f-18c383e81fc7-original-pull-secret\") pod \"global-pull-secret-syncer-fxlr6\" (UID: \"25a48584-b83e-4cff-a68f-18c383e81fc7\") " pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:54.390961 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.390937 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxlr6" Jul 16 11:04:54.472537 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.472509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w6bg\" (UniqueName: \"kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg\") pod \"network-check-target-c7mlk\" (UID: \"4055addb-97de-4ae7-b8e3-296404d93869\") " pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:54.472729 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.472558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs\") pod \"network-metrics-daemon-dvxhb\" (UID: \"2e52bd54-2b39-43e6-8494-8b74da851f0f\") " pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:54.476151 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.476123 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 11:04:54.476282 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.476243 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 11:04:54.485833 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.485797 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 11:04:54.486520 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.486485 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e52bd54-2b39-43e6-8494-8b74da851f0f-metrics-certs\") pod \"network-metrics-daemon-dvxhb\" (UID: \"2e52bd54-2b39-43e6-8494-8b74da851f0f\") " pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:54.496901 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.496827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w6bg\" (UniqueName: \"kubernetes.io/projected/4055addb-97de-4ae7-b8e3-296404d93869-kube-api-access-8w6bg\") pod \"network-check-target-c7mlk\" (UID: \"4055addb-97de-4ae7-b8e3-296404d93869\") " pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:54.533593 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.533538 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fxlr6"] Jul 16 11:04:54.698970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.698883 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-49sx5\"" Jul 16 11:04:54.702937 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.702910 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rdfxb\"" Jul 16 11:04:54.707027 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.707001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dvxhb" Jul 16 11:04:54.710977 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:54.710959 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:04:55.831437 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.831401 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5"] Jul 16 11:04:55.845748 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.845705 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:55.850071 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.850046 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 16 11:04:55.851492 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.851469 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-v96d6\"" Jul 16 11:04:55.851642 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.851615 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 11:04:55.851706 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.851642 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 11:04:55.851796 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.851719 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 16 11:04:55.851921 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.851908 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 11:04:55.861374 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.861293 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5"] Jul 16 11:04:55.882221 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.882133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8fa8508-ba77-46f9-b371-c935f10d0740-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-qx9k5\" (UID: \"d8fa8508-ba77-46f9-b371-c935f10d0740\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:55.882221 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.882199 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8fa8508-ba77-46f9-b371-c935f10d0740-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-qx9k5\" (UID: \"d8fa8508-ba77-46f9-b371-c935f10d0740\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:55.882221 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.882221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2xj6\" (UniqueName: \"kubernetes.io/projected/d8fa8508-ba77-46f9-b371-c935f10d0740-kube-api-access-m2xj6\") pod \"openshift-state-metrics-84bd68b896-qx9k5\" (UID: \"d8fa8508-ba77-46f9-b371-c935f10d0740\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:55.882458 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.882257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8fa8508-ba77-46f9-b371-c935f10d0740-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-qx9k5\" (UID: \"d8fa8508-ba77-46f9-b371-c935f10d0740\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:55.887094 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.887071 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-5sqjf"] Jul 16 11:04:55.890118 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.890097 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:55.893093 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.893076 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8twdc\"" Jul 16 11:04:55.893251 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.893234 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 16 11:04:55.893688 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.893671 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 16 11:04:55.893841 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.893824 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 16 11:04:55.914609 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.914500 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-5sqjf"] Jul 16 11:04:55.930021 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.929985 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4j8r2"] Jul 16 11:04:55.933614 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.933591 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:55.939979 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.939955 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 11:04:55.941255 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.941236 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-898zm\"" Jul 16 11:04:55.949458 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.949435 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 11:04:55.949601 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.949464 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 11:04:55.982870 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.982831 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vldcw\" (UniqueName: \"kubernetes.io/projected/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-kube-api-access-vldcw\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:55.983052 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.982877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-metrics-client-ca\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:55.983052 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.982916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8fa8508-ba77-46f9-b371-c935f10d0740-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-qx9k5\" (UID: \"d8fa8508-ba77-46f9-b371-c935f10d0740\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:55.983052 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.982943 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-wtmp\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:55.983052 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.982976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8fa8508-ba77-46f9-b371-c935f10d0740-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-qx9k5\" (UID: \"d8fa8508-ba77-46f9-b371-c935f10d0740\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:55.983052 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.983000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8fa8508-ba77-46f9-b371-c935f10d0740-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-qx9k5\" (UID: \"d8fa8508-ba77-46f9-b371-c935f10d0740\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:55.983052 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.983026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bb8e1f89-1dc4-49b7-9b40-980230ee378f-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:55.983052 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.983053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:55.983394 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:55.983110 2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 16 11:04:55.983394 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.983137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2xj6\" (UniqueName: \"kubernetes.io/projected/d8fa8508-ba77-46f9-b371-c935f10d0740-kube-api-access-m2xj6\") pod \"openshift-state-metrics-84bd68b896-qx9k5\" (UID: \"d8fa8508-ba77-46f9-b371-c935f10d0740\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:55.983394 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:55.983191 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8fa8508-ba77-46f9-b371-c935f10d0740-openshift-state-metrics-tls podName:d8fa8508-ba77-46f9-b371-c935f10d0740 nodeName:}" failed. No retries permitted until 2026-07-16 11:04:56.483168843 +0000 UTC m=+68.316517524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d8fa8508-ba77-46f9-b371-c935f10d0740-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-qx9k5" (UID: "d8fa8508-ba77-46f9-b371-c935f10d0740") : secret "openshift-state-metrics-tls" not found Jul 16 11:04:55.983394 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.983216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:55.983394 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.983248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb8e1f89-1dc4-49b7-9b40-980230ee378f-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:55.983394 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.983284 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-textfile\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:55.983394 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.983335 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-tls\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:55.983394 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.983372 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:55.983394 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.983406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-sys\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:55.983888 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.983427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-root\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:55.983888 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.983447 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:55.983888 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.983485 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4lkf\" (UniqueName: \"kubernetes.io/projected/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-api-access-t4lkf\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:55.983888 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.983519 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:55.984084 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.984017 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8fa8508-ba77-46f9-b371-c935f10d0740-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-qx9k5\" (UID: \"d8fa8508-ba77-46f9-b371-c935f10d0740\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:55.985924 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:55.985897 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8fa8508-ba77-46f9-b371-c935f10d0740-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-qx9k5\" (UID: \"d8fa8508-ba77-46f9-b371-c935f10d0740\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:56.014768 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.014708 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2xj6\" (UniqueName: \"kubernetes.io/projected/d8fa8508-ba77-46f9-b371-c935f10d0740-kube-api-access-m2xj6\") pod \"openshift-state-metrics-84bd68b896-qx9k5\" (UID: \"d8fa8508-ba77-46f9-b371-c935f10d0740\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:56.084217 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:56.084217 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084176 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb8e1f89-1dc4-49b7-9b40-980230ee378f-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:56.084217 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-textfile\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.084217 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-tls\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.084531 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:56.084531 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-sys\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.084531 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:56.084310 2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 16 11:04:56.084531 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084359 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-sys\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.084531 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:04:56.084379 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-state-metrics-tls podName:bb8e1f89-1dc4-49b7-9b40-980230ee378f nodeName:}" failed. No retries permitted until 2026-07-16 11:04:56.584357819 +0000 UTC m=+68.417706479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-5sqjf" (UID: "bb8e1f89-1dc4-49b7-9b40-980230ee378f") : secret "kube-state-metrics-tls" not found Jul 16 11:04:56.084531 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-root\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.084531 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084444 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:56.084531 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4lkf\" (UniqueName: \"kubernetes.io/projected/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-api-access-t4lkf\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:56.084531 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.084963 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vldcw\" (UniqueName: \"kubernetes.io/projected/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-kube-api-access-vldcw\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.084963 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-metrics-client-ca\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.084963 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-wtmp\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.084963 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bb8e1f89-1dc4-49b7-9b40-980230ee378f-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:56.084963 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.084963 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.084953 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-root\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.085228 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.085026 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-wtmp\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.085228 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.085063 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb8e1f89-1dc4-49b7-9b40-980230ee378f-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:56.085372 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.085325 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bb8e1f89-1dc4-49b7-9b40-980230ee378f-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:56.085794 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.085544 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-textfile\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.085794 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.085640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:56.085985 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.085957 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-metrics-client-ca\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.086088 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.086067 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.087057 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.087035 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-tls\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.087304 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.087285 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:56.087908 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.087887 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.095141 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.095121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vldcw\" (UniqueName: \"kubernetes.io/projected/184d37e6-e16f-4f5f-9c83-c423c8c53d4a-kube-api-access-vldcw\") pod \"node-exporter-4j8r2\" (UID: \"184d37e6-e16f-4f5f-9c83-c423c8c53d4a\") " pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.098687 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.098664 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4lkf\" (UniqueName: \"kubernetes.io/projected/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-api-access-t4lkf\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:56.148330 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:04:56.148288 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a48584_b83e_4cff_a68f_18c383e81fc7.slice/crio-5be183ac80a492451dc10d7148e5e8116d9db0fc1714220351b8bee0238dc976 WatchSource:0}: Error finding container 5be183ac80a492451dc10d7148e5e8116d9db0fc1714220351b8bee0238dc976: Status 404 returned error can't find the container with id 5be183ac80a492451dc10d7148e5e8116d9db0fc1714220351b8bee0238dc976 Jul 16 11:04:56.243313 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.243205 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4j8r2" Jul 16 11:04:56.259061 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:04:56.259016 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184d37e6_e16f_4f5f_9c83_c423c8c53d4a.slice/crio-e9f523cbe66a150474137be99ec403e2574d71ce1f1c4bf4b592a978b439ee42 WatchSource:0}: Error finding container e9f523cbe66a150474137be99ec403e2574d71ce1f1c4bf4b592a978b439ee42: Status 404 returned error can't find the container with id e9f523cbe66a150474137be99ec403e2574d71ce1f1c4bf4b592a978b439ee42 Jul 16 11:04:56.286995 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.286960 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c7mlk"] Jul 16 11:04:56.290690 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:04:56.290662 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4055addb_97de_4ae7_b8e3_296404d93869.slice/crio-ce68d2f000ccce34779694b54ff3efed72817119ca08aed15691115f2377705d WatchSource:0}: Error finding container ce68d2f000ccce34779694b54ff3efed72817119ca08aed15691115f2377705d: Status 404 returned error can't find the container with id ce68d2f000ccce34779694b54ff3efed72817119ca08aed15691115f2377705d Jul 16 11:04:56.305846 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.305817 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dvxhb"] Jul 16 11:04:56.308990 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:04:56.308964 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e52bd54_2b39_43e6_8494_8b74da851f0f.slice/crio-6571947b97ceafcc207e660721bfae2cf68b01d0dc7416f4ff1c4bf3f3400ab4 WatchSource:0}: Error finding container 6571947b97ceafcc207e660721bfae2cf68b01d0dc7416f4ff1c4bf3f3400ab4: Status 404 returned error can't find the container with id 6571947b97ceafcc207e660721bfae2cf68b01d0dc7416f4ff1c4bf3f3400ab4 Jul 16 11:04:56.488597 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.488491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8fa8508-ba77-46f9-b371-c935f10d0740-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-qx9k5\" (UID: \"d8fa8508-ba77-46f9-b371-c935f10d0740\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:56.490824 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.490805 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8fa8508-ba77-46f9-b371-c935f10d0740-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-qx9k5\" (UID: \"d8fa8508-ba77-46f9-b371-c935f10d0740\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:56.589959 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.589919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:56.592500 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.592472 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb8e1f89-1dc4-49b7-9b40-980230ee378f-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5sqjf\" (UID: \"bb8e1f89-1dc4-49b7-9b40-980230ee378f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:56.757960 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.757929 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" Jul 16 11:04:56.801189 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.801151 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" Jul 16 11:04:56.889090 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.888041 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 11:04:56.901588 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.901531 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:56.913092 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.913052 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 11:04:56.913547 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.913529 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 11:04:56.918126 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.913847 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 11:04:56.918126 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.913869 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 11:04:56.918126 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.914077 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 11:04:56.918126 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.914156 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 11:04:56.918126 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.916040 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 11:04:56.918126 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.916257 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qmsx2\"" Jul 16 11:04:56.918126 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.916439 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 11:04:56.918126 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.916710 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 11:04:56.918126 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.916944 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 11:04:56.950480 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:56.950450 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5"] Jul 16 11:04:57.002070 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.001978 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-5sqjf"] Jul 16 11:04:57.007440 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:04:57.007411 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8e1f89_1dc4_49b7_9b40_980230ee378f.slice/crio-22eade567d268eb2e0358937533f17068a5fc038a79b901cf538412ae36c46a7 WatchSource:0}: Error finding container 22eade567d268eb2e0358937533f17068a5fc038a79b901cf538412ae36c46a7: Status 404 returned error can't find the container with id 22eade567d268eb2e0358937533f17068a5fc038a79b901cf538412ae36c46a7 Jul 16 11:04:57.013116 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.012914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.013116 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.012952 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.013474 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.013383 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.013809 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.013667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.013809 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.013708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-config-out\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.013809 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.013735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-web-config\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.014646 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.014126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-config-volume\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.014646 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.014178 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.014646 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.014213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.014646 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.014256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.014646 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.014296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.014646 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.014352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.014646 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.014380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg2lf\" (UniqueName: \"kubernetes.io/projected/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-kube-api-access-rg2lf\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.115722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.115146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.115722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.115210 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.115722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.115240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.115722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.115265 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg2lf\" (UniqueName: \"kubernetes.io/projected/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-kube-api-access-rg2lf\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.115722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.115311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.115722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.115337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.115722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.115371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.115722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.115394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.115722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.115419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-config-out\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.115722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.115447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-web-config\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.115722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.115479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-config-volume\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.115722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.115526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.115722 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.115559 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.117313 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.117112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.117589 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.117548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.118921 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.118895 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.124395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.123941 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-config-out\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.124395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.124358 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-config-volume\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.125220 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.125178 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.125762 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.125709 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.126763 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.126738 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg2lf\" (UniqueName: \"kubernetes.io/projected/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-kube-api-access-rg2lf\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.128788 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.128769 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.130378 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.129340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.130378 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.130023 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.130378 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.130316 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.131239 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.131199 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f0f0c71-ac01-4c43-8464-9e8c4bd2056a-web-config\") pod \"alertmanager-main-0\" (UID: \"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:57.169477 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.169268 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nxfs5" event={"ID":"00196644-3e96-4e0a-9002-ac8d51e53cc1","Type":"ContainerStarted","Data":"cac7414eaffa3392f4e635c98b1dcbc7d55e753a8c1a25806957fefac2f412f2"} Jul 16 11:04:57.171769 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.171739 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" event={"ID":"bb8e1f89-1dc4-49b7-9b40-980230ee378f","Type":"ContainerStarted","Data":"22eade567d268eb2e0358937533f17068a5fc038a79b901cf538412ae36c46a7"} Jul 16 11:04:57.176408 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.176348 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c7mlk" event={"ID":"4055addb-97de-4ae7-b8e3-296404d93869","Type":"ContainerStarted","Data":"ce68d2f000ccce34779694b54ff3efed72817119ca08aed15691115f2377705d"} Jul 16 11:04:57.178733 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.178679 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fxlr6" event={"ID":"25a48584-b83e-4cff-a68f-18c383e81fc7","Type":"ContainerStarted","Data":"5be183ac80a492451dc10d7148e5e8116d9db0fc1714220351b8bee0238dc976"} Jul 16 11:04:57.181764 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.181737 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" event={"ID":"d8fa8508-ba77-46f9-b371-c935f10d0740","Type":"ContainerStarted","Data":"7f614f8cc5431e9096afab9cb2d9e19c27ba06c14e8c33af7cf628f52be501de"} Jul 16 11:04:57.181866 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.181797 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" event={"ID":"d8fa8508-ba77-46f9-b371-c935f10d0740","Type":"ContainerStarted","Data":"191c1834abfd85660f319649bf2714ddc48e7146cb9c958c0b3e2f8b0eccabc4"} Jul 16 11:04:57.184244 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.184216 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4j8r2" event={"ID":"184d37e6-e16f-4f5f-9c83-c423c8c53d4a","Type":"ContainerStarted","Data":"e9f523cbe66a150474137be99ec403e2574d71ce1f1c4bf4b592a978b439ee42"} Jul 16 11:04:57.186471 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.186446 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dvxhb" event={"ID":"2e52bd54-2b39-43e6-8494-8b74da851f0f","Type":"ContainerStarted","Data":"6571947b97ceafcc207e660721bfae2cf68b01d0dc7416f4ff1c4bf3f3400ab4"} Jul 16 11:04:57.190533 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.190482 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nxfs5" podStartSLOduration=1.815951486 podStartE2EDuration="17.190462458s" totalImagesPullingTime="15.374510972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:04:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:57.188291701 +0000 UTC m=+69.021640386" watchObservedRunningTime="2026-07-16 11:04:57.190462458 +0000 UTC m=+69.023811153" Jul 16 11:04:57.232104 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:57.232037 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:04:58.194605 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:58.194555 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" event={"ID":"d8fa8508-ba77-46f9-b371-c935f10d0740","Type":"ContainerStarted","Data":"df15d4b84a07c5b84d06274612444536531b52e6f0f80959c06f625fb4dddce1"} Jul 16 11:04:58.449077 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:58.447855 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 11:04:59.131141 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:59.131114 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ddhln" Jul 16 11:04:59.201052 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:59.201014 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dvxhb" event={"ID":"2e52bd54-2b39-43e6-8494-8b74da851f0f","Type":"ContainerStarted","Data":"69e345a97df76c8e83bf1f41c6e790e00481a38b03893dd099befbd30048dc99"} Jul 16 11:04:59.201478 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:59.201084 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dvxhb" event={"ID":"2e52bd54-2b39-43e6-8494-8b74da851f0f","Type":"ContainerStarted","Data":"b070eb3fb8e1f38412c515d751c38c693e6aed8e66c731607d5c443ad93d439e"} Jul 16 11:04:59.202853 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:59.202819 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a","Type":"ContainerStarted","Data":"ac386eb02b0fb183060c01cd16ac6589210269bc3d8c1213a9cb481d485d11af"} Jul 16 11:04:59.204713 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:59.204686 2580 generic.go:356] "Generic (PLEG): container finished" podID="184d37e6-e16f-4f5f-9c83-c423c8c53d4a" containerID="0a394b292b1392bb99ba588581a47841ff124250b6a27f6885f414fa6d228f71" exitCode=0 Jul 16 11:04:59.204819 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:59.204725 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4j8r2" event={"ID":"184d37e6-e16f-4f5f-9c83-c423c8c53d4a","Type":"ContainerDied","Data":"0a394b292b1392bb99ba588581a47841ff124250b6a27f6885f414fa6d228f71"} Jul 16 11:04:59.249692 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:04:59.249624 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dvxhb" podStartSLOduration=68.294869864 podStartE2EDuration="1m10.249605001s" totalImagesPullingTime="1.954735137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:04:59.222526192 +0000 UTC m=+71.055874876" watchObservedRunningTime="2026-07-16 11:04:59.249605001 +0000 UTC m=+71.082953689" Jul 16 11:05:00.250709 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.250673 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-76c4dbdb5b-jhthw"] Jul 16 11:05:00.253455 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.253425 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.257634 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.257605 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cqk5gbdgmkrr9\"" Jul 16 11:05:00.257772 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.257634 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bsqbq\"" Jul 16 11:05:00.257772 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.257634 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 16 11:05:00.257863 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.257836 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 11:05:00.257918 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.257869 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 16 11:05:00.257993 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.257962 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 16 11:05:00.264963 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.264940 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76c4dbdb5b-jhthw"] Jul 16 11:05:00.348242 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.348210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrhjm\" (UniqueName: \"kubernetes.io/projected/347220ac-836a-4ec0-8036-d639a8a494fd-kube-api-access-nrhjm\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.348408 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.348275 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/347220ac-836a-4ec0-8036-d639a8a494fd-secret-metrics-server-tls\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.348408 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.348312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/347220ac-836a-4ec0-8036-d639a8a494fd-metrics-server-audit-profiles\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.348408 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.348376 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/347220ac-836a-4ec0-8036-d639a8a494fd-secret-metrics-server-client-certs\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.348621 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.348408 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/347220ac-836a-4ec0-8036-d639a8a494fd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.348621 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.348436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/347220ac-836a-4ec0-8036-d639a8a494fd-audit-log\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.348621 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.348461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/347220ac-836a-4ec0-8036-d639a8a494fd-client-ca-bundle\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.449657 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.449617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/347220ac-836a-4ec0-8036-d639a8a494fd-secret-metrics-server-tls\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.449834 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.449680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/347220ac-836a-4ec0-8036-d639a8a494fd-metrics-server-audit-profiles\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.449834 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.449735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/347220ac-836a-4ec0-8036-d639a8a494fd-secret-metrics-server-client-certs\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.449834 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.449769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/347220ac-836a-4ec0-8036-d639a8a494fd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.449834 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.449799 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/347220ac-836a-4ec0-8036-d639a8a494fd-audit-log\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.449834 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.449823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/347220ac-836a-4ec0-8036-d639a8a494fd-client-ca-bundle\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.450089 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.449873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrhjm\" (UniqueName: \"kubernetes.io/projected/347220ac-836a-4ec0-8036-d639a8a494fd-kube-api-access-nrhjm\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.451678 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.450738 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/347220ac-836a-4ec0-8036-d639a8a494fd-audit-log\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.451678 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.450860 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/347220ac-836a-4ec0-8036-d639a8a494fd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.451678 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.451612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/347220ac-836a-4ec0-8036-d639a8a494fd-metrics-server-audit-profiles\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.454139 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.454093 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/347220ac-836a-4ec0-8036-d639a8a494fd-secret-metrics-server-tls\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.454535 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.454502 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/347220ac-836a-4ec0-8036-d639a8a494fd-client-ca-bundle\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.454948 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.454926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/347220ac-836a-4ec0-8036-d639a8a494fd-secret-metrics-server-client-certs\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.459759 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.459734 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrhjm\" (UniqueName: \"kubernetes.io/projected/347220ac-836a-4ec0-8036-d639a8a494fd-kube-api-access-nrhjm\") pod \"metrics-server-76c4dbdb5b-jhthw\" (UID: \"347220ac-836a-4ec0-8036-d639a8a494fd\") " pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.567908 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.567790 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:00.643379 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.643345 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bc8dc6588-746qt"] Jul 16 11:05:00.645816 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.645786 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.661406 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.661338 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bc8dc6588-746qt"] Jul 16 11:05:00.752549 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.752510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-oauth-serving-cert\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.752549 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.752551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb6lp\" (UniqueName: \"kubernetes.io/projected/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-kube-api-access-zb6lp\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.752810 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.752617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-serving-cert\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.752810 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.752667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-config\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.752810 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.752696 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-trusted-ca-bundle\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.752954 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.752830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-oauth-config\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.752954 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.752889 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-service-ca\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.853862 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.853776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-config\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.853862 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.853816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-trusted-ca-bundle\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.854117 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.853879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-oauth-config\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.854117 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.853901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-service-ca\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.854117 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.853929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-oauth-serving-cert\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.854117 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.853954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb6lp\" (UniqueName: \"kubernetes.io/projected/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-kube-api-access-zb6lp\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.854117 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.854010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-serving-cert\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.854718 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.854688 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-config\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.854838 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.854781 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-service-ca\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.854902 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.854867 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-oauth-serving-cert\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.855005 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.854911 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-trusted-ca-bundle\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.856763 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.856723 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-serving-cert\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.857177 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.857148 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-oauth-config\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.863587 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.863542 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb6lp\" (UniqueName: \"kubernetes.io/projected/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-kube-api-access-zb6lp\") pod \"console-6bc8dc6588-746qt\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:00.958292 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:00.958244 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:01.648917 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:01.648885 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:05:01.653236 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:01.653214 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:05:02.213414 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.212122 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 11:05:02.226046 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.226015 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4j8r2" event={"ID":"184d37e6-e16f-4f5f-9c83-c423c8c53d4a","Type":"ContainerStarted","Data":"fcbe51b71349f4ce43e1a96454f37207d18485453b6fae755e127eeb1e8bd8cd"} Jul 16 11:05:02.226359 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.226200 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.231484 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.231312 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 11:05:02.231484 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.231361 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bwwb2\"" Jul 16 11:05:02.231825 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.231637 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 11:05:02.231825 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.231681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c7mlk" event={"ID":"4055addb-97de-4ae7-b8e3-296404d93869","Type":"ContainerStarted","Data":"c00690f0654f66e8c51579bb923135c914e69141b2ceb8b8c064a0780b018ede"} Jul 16 11:05:02.231825 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.231714 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 11:05:02.231825 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.231759 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:05:02.233391 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.232441 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 11:05:02.233391 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.232483 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 11:05:02.233391 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.232603 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 11:05:02.233391 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.232440 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1dr2bo6nvjv6n\"" Jul 16 11:05:02.233391 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.232877 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 11:05:02.233391 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.232975 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 11:05:02.233391 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.233151 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 11:05:02.233391 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.233219 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 11:05:02.239379 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.235224 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 11:05:02.239379 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.237105 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 11:05:02.239379 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.237743 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 11:05:02.239379 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.237742 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 11:05:02.244165 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.244145 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 11:05:02.244355 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.244341 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76c4dbdb5b-jhthw"] Jul 16 11:05:02.248209 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:05:02.248180 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347220ac_836a_4ec0_8036_d639a8a494fd.slice/crio-79e36eea1469d13f3a1587840d4eba833c280aa2bf0d8a564c2fe4de3d103e4f WatchSource:0}: Error finding container 79e36eea1469d13f3a1587840d4eba833c280aa2bf0d8a564c2fe4de3d103e4f: Status 404 returned error can't find the container with id 79e36eea1469d13f3a1587840d4eba833c280aa2bf0d8a564c2fe4de3d103e4f Jul 16 11:05:02.277794 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.277721 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bc8dc6588-746qt"] Jul 16 11:05:02.299982 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.299942 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c7mlk" podStartSLOduration=67.551278848 podStartE2EDuration="1m13.29992141s" totalImagesPullingTime="5.748642562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:05:02.297392784 +0000 UTC m=+74.130741466" watchObservedRunningTime="2026-07-16 11:05:02.29992141 +0000 UTC m=+74.133270095" Jul 16 11:05:02.375877 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.375834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.375877 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.375874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376067 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.375905 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376067 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.375933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtmpb\" (UniqueName: \"kubernetes.io/projected/31bfd179-2649-4908-ab65-6856930379bc-kube-api-access-qtmpb\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376067 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.375976 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376067 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376259 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376078 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376259 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376259 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376259 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376259 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-web-config\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376491 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31bfd179-2649-4908-ab65-6856930379bc-config-out\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376491 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376281 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376491 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376335 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31bfd179-2649-4908-ab65-6856930379bc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376491 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376491 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376389 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376491 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376472 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376886 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376886 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376538 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31bfd179-2649-4908-ab65-6856930379bc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.376886 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.376669 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-config\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.477432 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477399 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.477432 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtmpb\" (UniqueName: \"kubernetes.io/projected/31bfd179-2649-4908-ab65-6856930379bc-kube-api-access-qtmpb\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.477730 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.477730 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.477730 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.477730 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.477730 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.477730 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477610 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.477730 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-web-config\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.477730 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477654 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31bfd179-2649-4908-ab65-6856930379bc-config-out\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.477730 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.477730 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31bfd179-2649-4908-ab65-6856930379bc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.477730 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.477730 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.478330 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.478330 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.478330 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477804 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31bfd179-2649-4908-ab65-6856930379bc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.478330 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-config\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.478330 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.478330 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.477867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.478998 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.478969 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.479468 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.479416 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.479682 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.479663 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31bfd179-2649-4908-ab65-6856930379bc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.479803 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.479778 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.480936 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.480902 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.481103 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.481072 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.482169 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.482128 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.483597 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.483534 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.483705 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.483656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.484049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.483792 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.484049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.483970 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31bfd179-2649-4908-ab65-6856930379bc-config-out\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.484179 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.484145 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31bfd179-2649-4908-ab65-6856930379bc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.485285 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.484724 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31bfd179-2649-4908-ab65-6856930379bc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.485285 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.484989 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.485285 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.485247 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.485285 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.485266 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-config\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.485910 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.485888 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-web-config\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.487457 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.486990 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.487457 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.487404 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31bfd179-2649-4908-ab65-6856930379bc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.489030 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.488944 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtmpb\" (UniqueName: \"kubernetes.io/projected/31bfd179-2649-4908-ab65-6856930379bc-kube-api-access-qtmpb\") pod \"prometheus-k8s-0\" (UID: \"31bfd179-2649-4908-ab65-6856930379bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.569629 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.569472 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:02.724720 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:02.724660 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 11:05:02.727167 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:05:02.727138 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bfd179_2649_4908_ab65_6856930379bc.slice/crio-6fb935e9453e692ad5e6c22184ee3e69a52444d4ed308e9df22f8fc58946b2cd WatchSource:0}: Error finding container 6fb935e9453e692ad5e6c22184ee3e69a52444d4ed308e9df22f8fc58946b2cd: Status 404 returned error can't find the container with id 6fb935e9453e692ad5e6c22184ee3e69a52444d4ed308e9df22f8fc58946b2cd Jul 16 11:05:03.237981 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.237886 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" event={"ID":"d8fa8508-ba77-46f9-b371-c935f10d0740","Type":"ContainerStarted","Data":"75cf6d33595f3d8850f44d06bc4f19fa987bde3c5e7fabf2157a5bfa6d05fd98"} Jul 16 11:05:03.240374 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.240346 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4j8r2" event={"ID":"184d37e6-e16f-4f5f-9c83-c423c8c53d4a","Type":"ContainerStarted","Data":"25277fe114c57bbf8ad9c71b6fa565d74f6ae1522aa3f376045646af8e063494"} Jul 16 11:05:03.246680 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.246649 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc8dc6588-746qt" event={"ID":"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7","Type":"ContainerStarted","Data":"a9b9050d4ed44d04d0c1483634ed2499e7edec75a4ac851cf696d8b053d43df3"} Jul 16 11:05:03.246903 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.246876 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc8dc6588-746qt" event={"ID":"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7","Type":"ContainerStarted","Data":"28da74c5d177e9a790da56713d7df7fe7f690b7d3c48477937f941576c88014a"} Jul 16 11:05:03.248051 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.248028 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" event={"ID":"347220ac-836a-4ec0-8036-d639a8a494fd","Type":"ContainerStarted","Data":"79e36eea1469d13f3a1587840d4eba833c280aa2bf0d8a564c2fe4de3d103e4f"} Jul 16 11:05:03.250336 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.250313 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" event={"ID":"bb8e1f89-1dc4-49b7-9b40-980230ee378f","Type":"ContainerStarted","Data":"14919168f886671478faffab0f6a7cb28e4dcf61166302d7eb68a24997f89142"} Jul 16 11:05:03.250447 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.250362 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" event={"ID":"bb8e1f89-1dc4-49b7-9b40-980230ee378f","Type":"ContainerStarted","Data":"0f1a5b2960e45a140cc26ad289750c580252c564e011c7a2379eabf76978b9a2"} Jul 16 11:05:03.250447 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.250392 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" event={"ID":"bb8e1f89-1dc4-49b7-9b40-980230ee378f","Type":"ContainerStarted","Data":"0c47ae94baed82bab91e0a8033bb3d262fb7a454ab54f84badc6ddd2d17e2e9e"} Jul 16 11:05:03.251849 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.251824 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fxlr6" event={"ID":"25a48584-b83e-4cff-a68f-18c383e81fc7","Type":"ContainerStarted","Data":"18c0196ef4be2216b9f4b5b4e28fea05c075f01c77907ee89a8beb36a7d2c7e6"} Jul 16 11:05:03.253334 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.253309 2580 generic.go:356] "Generic (PLEG): container finished" podID="7f0f0c71-ac01-4c43-8464-9e8c4bd2056a" containerID="22c90b7e6fd67c21fc1d94af13f9196c506dcd761914dacf1151acf5651c9c8b" exitCode=0 Jul 16 11:05:03.253444 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.253392 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a","Type":"ContainerDied","Data":"22c90b7e6fd67c21fc1d94af13f9196c506dcd761914dacf1151acf5651c9c8b"} Jul 16 11:05:03.255278 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.255255 2580 generic.go:356] "Generic (PLEG): container finished" podID="31bfd179-2649-4908-ab65-6856930379bc" containerID="ec2fdd8d84b54b7856829614b53c7ee90bc58ca393f83016ccb1aa508821e926" exitCode=0 Jul 16 11:05:03.255375 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.255355 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31bfd179-2649-4908-ab65-6856930379bc","Type":"ContainerDied","Data":"ec2fdd8d84b54b7856829614b53c7ee90bc58ca393f83016ccb1aa508821e926"} Jul 16 11:05:03.255422 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.255411 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31bfd179-2649-4908-ab65-6856930379bc","Type":"ContainerStarted","Data":"6fb935e9453e692ad5e6c22184ee3e69a52444d4ed308e9df22f8fc58946b2cd"} Jul 16 11:05:03.261517 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.261469 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qx9k5" podStartSLOduration=3.42903914 podStartE2EDuration="8.261454866s" totalImagesPullingTime="4.832415726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:05:03.260283074 +0000 UTC m=+75.093631759" watchObservedRunningTime="2026-07-16 11:05:03.261454866 +0000 UTC m=+75.094803559" Jul 16 11:05:03.279638 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.279563 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-5sqjf" podStartSLOduration=3.248130506 podStartE2EDuration="8.279544161s" totalImagesPullingTime="5.031413655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:05:03.279143776 +0000 UTC m=+75.112492462" watchObservedRunningTime="2026-07-16 11:05:03.279544161 +0000 UTC m=+75.112892876" Jul 16 11:05:03.299949 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.299891 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4j8r2" podStartSLOduration=6.164396081 podStartE2EDuration="8.299875274s" totalImagesPullingTime="1.997567806s" totalInitContainerRuntime="137.911387ms" isStatefulPod=true podCreationTimestamp="2026-07-16 11:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:05:03.29888344 +0000 UTC m=+75.132232147" watchObservedRunningTime="2026-07-16 11:05:03.299875274 +0000 UTC m=+75.133223956" Jul 16 11:05:03.381488 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.381435 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bc8dc6588-746qt" podStartSLOduration=3.38141189 podStartE2EDuration="3.38141189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:05:03.379792135 +0000 UTC m=+75.213140818" watchObservedRunningTime="2026-07-16 11:05:03.38141189 +0000 UTC m=+75.214760553" Jul 16 11:05:03.404564 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:03.404502 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fxlr6" podStartSLOduration=68.513920863 podStartE2EDuration="1m14.404480087s" totalImagesPullingTime="5.890559224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:05:03.402125275 +0000 UTC m=+75.235473960" watchObservedRunningTime="2026-07-16 11:05:03.404480087 +0000 UTC m=+75.237828772" Jul 16 11:05:04.261865 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:04.261823 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" event={"ID":"347220ac-836a-4ec0-8036-d639a8a494fd","Type":"ContainerStarted","Data":"d67aca0bf7bb26e211031b17a8d09c0c10e68ecfd8cd067f10c755bfde2fd780"} Jul 16 11:05:04.283275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:04.283205 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" podStartSLOduration=2.947593139 podStartE2EDuration="4.28318197s" totalImagesPullingTime="1.335588831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:05:04.282092811 +0000 UTC m=+76.115441485" watchObservedRunningTime="2026-07-16 11:05:04.28318197 +0000 UTC m=+76.116530654" Jul 16 11:05:05.269856 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:05.269823 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a","Type":"ContainerStarted","Data":"ae5dc184fe2963659706e86ad52418778a505fa49003b12f41415bd8fad45129"} Jul 16 11:05:05.270263 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:05.269894 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a","Type":"ContainerStarted","Data":"4b066afc340622941aabab7c49d8c04700887699bfe8bc8ea6007e3e863a3993"} Jul 16 11:05:05.270263 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:05.269922 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a","Type":"ContainerStarted","Data":"1852279ea19561b9403dec5bf02d71fe7f4407283ccf29ba3e247857113c03e7"} Jul 16 11:05:05.270263 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:05.269945 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a","Type":"ContainerStarted","Data":"8df3606d7f78d7c9ec6c6caf02a731adf9392c01cea6087a571ed08a9ef79735"} Jul 16 11:05:05.270263 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:05.269984 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a","Type":"ContainerStarted","Data":"8f5e68160235e99007ce2d08179e8c7c3e6e9def7441c261c9a5f4780392af3f"} Jul 16 11:05:07.282285 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:07.282240 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f0f0c71-ac01-4c43-8464-9e8c4bd2056a","Type":"ContainerStarted","Data":"66130ae6797d760af72ba0d8dc26558cd87f39d2c250615474581413f20771ef"} Jul 16 11:05:07.284847 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:07.284820 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31bfd179-2649-4908-ab65-6856930379bc","Type":"ContainerStarted","Data":"0d3085a66d2fb0503c3196d9b6cde5c3c49f39b8daa589cda92d8a2bfecfba1c"} Jul 16 11:05:07.284957 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:07.284881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31bfd179-2649-4908-ab65-6856930379bc","Type":"ContainerStarted","Data":"b00a6c6a79cb81cc6bd0540d6a7bb0000789a07733b9a463500f9888e27b2bdb"} Jul 16 11:05:07.316908 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:07.316857 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.345425826 podStartE2EDuration="11.316836587s" totalImagesPullingTime="6.949775874s" totalInitContainerRuntime="21.634887ms" isStatefulPod=false podCreationTimestamp="2026-07-16 11:04:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:05:07.314459628 +0000 UTC m=+79.147808311" watchObservedRunningTime="2026-07-16 11:05:07.316836587 +0000 UTC m=+79.150185273" Jul 16 11:05:09.296377 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:09.296339 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31bfd179-2649-4908-ab65-6856930379bc","Type":"ContainerStarted","Data":"d25c8e539643be98752baadd320ead9a3d36d92463f5035409fc8ddeb348f3d7"} Jul 16 11:05:09.296744 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:09.296421 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31bfd179-2649-4908-ab65-6856930379bc","Type":"ContainerStarted","Data":"def4c400af2a6e1fd7b2ba814588f15907339a995d62ae5b6567cd178ddb6c97"} Jul 16 11:05:09.296744 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:09.296467 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31bfd179-2649-4908-ab65-6856930379bc","Type":"ContainerStarted","Data":"5d4f87cb2277831ea9c964bb5b7b83cd37b13d732dce8516accec225cae0ce3a"} Jul 16 11:05:09.296744 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:09.296502 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31bfd179-2649-4908-ab65-6856930379bc","Type":"ContainerStarted","Data":"e824b438273fd1bcff97fee504f23aeb64004b4dd37cd08a3708435a112cb639"} Jul 16 11:05:09.333854 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:09.333789 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.291443673 podStartE2EDuration="7.333771341s" totalImagesPullingTime="5.013748703s" totalInitContainerRuntime="28.578965ms" isStatefulPod=false podCreationTimestamp="2026-07-16 11:05:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:05:09.331149626 +0000 UTC m=+81.164498307" watchObservedRunningTime="2026-07-16 11:05:09.333771341 +0000 UTC m=+81.167120027" Jul 16 11:05:10.958969 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:10.958857 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:10.958969 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:10.958923 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:10.963833 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:10.963811 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:11.307912 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:11.307883 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:05:11.393646 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:11.393613 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c76d5874-lsbs4"] Jul 16 11:05:12.570522 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:12.570473 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:05:20.568798 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:20.568756 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:20.569253 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:20.568834 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:33.258419 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:33.258389 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c7mlk" Jul 16 11:05:36.414349 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:36.414293 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c76d5874-lsbs4" podUID="b3f323d8-8779-4dcf-9e15-e1a55b70317d" containerName="console" containerID="cri-o://c476993975558f84715d4a1fa2e33311e3e6799dd418e32e2b5d213145016882" gracePeriod=15 Jul 16 11:05:37.186897 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.186873 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:05:37.305317 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.305284 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-oauth-config\") pod \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " Jul 16 11:05:37.305317 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.305322 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-config\") pod \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " Jul 16 11:05:37.305536 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.305352 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-serving-cert\") pod \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " Jul 16 11:05:37.305536 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.305384 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-oauth-serving-cert\") pod \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " Jul 16 11:05:37.305536 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.305439 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr8p2\" (UniqueName: \"kubernetes.io/projected/b3f323d8-8779-4dcf-9e15-e1a55b70317d-kube-api-access-fr8p2\") pod \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " Jul 16 11:05:37.305536 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.305467 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-trusted-ca-bundle\") pod \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " Jul 16 11:05:37.305536 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.305527 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-service-ca\") pod \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\" (UID: \"b3f323d8-8779-4dcf-9e15-e1a55b70317d\") " Jul 16 11:05:37.305797 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.305763 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-config" (OuterVolumeSpecName: "console-config") pod "b3f323d8-8779-4dcf-9e15-e1a55b70317d" (UID: "b3f323d8-8779-4dcf-9e15-e1a55b70317d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:05:37.305941 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.305917 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b3f323d8-8779-4dcf-9e15-e1a55b70317d" (UID: "b3f323d8-8779-4dcf-9e15-e1a55b70317d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:05:37.306062 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.306036 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b3f323d8-8779-4dcf-9e15-e1a55b70317d" (UID: "b3f323d8-8779-4dcf-9e15-e1a55b70317d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:05:37.306136 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.306113 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-service-ca" (OuterVolumeSpecName: "service-ca") pod "b3f323d8-8779-4dcf-9e15-e1a55b70317d" (UID: "b3f323d8-8779-4dcf-9e15-e1a55b70317d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:05:37.307844 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.307813 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b3f323d8-8779-4dcf-9e15-e1a55b70317d" (UID: "b3f323d8-8779-4dcf-9e15-e1a55b70317d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:05:37.307926 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.307882 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b3f323d8-8779-4dcf-9e15-e1a55b70317d" (UID: "b3f323d8-8779-4dcf-9e15-e1a55b70317d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:05:37.307926 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.307898 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f323d8-8779-4dcf-9e15-e1a55b70317d-kube-api-access-fr8p2" (OuterVolumeSpecName: "kube-api-access-fr8p2") pod "b3f323d8-8779-4dcf-9e15-e1a55b70317d" (UID: "b3f323d8-8779-4dcf-9e15-e1a55b70317d"). InnerVolumeSpecName "kube-api-access-fr8p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:05:37.406407 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.406331 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-oauth-config\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:05:37.406407 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.406360 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-config\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:05:37.406407 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.406370 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3f323d8-8779-4dcf-9e15-e1a55b70317d-console-serving-cert\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:05:37.406407 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.406400 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-oauth-serving-cert\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:05:37.406407 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.406409 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr8p2\" (UniqueName: \"kubernetes.io/projected/b3f323d8-8779-4dcf-9e15-e1a55b70317d-kube-api-access-fr8p2\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:05:37.406731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.406418 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-trusted-ca-bundle\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:05:37.406731 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.406427 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f323d8-8779-4dcf-9e15-e1a55b70317d-service-ca\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:05:37.415744 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.415714 2580 generic.go:356] "Generic (PLEG): container finished" podID="b3f323d8-8779-4dcf-9e15-e1a55b70317d" containerID="c476993975558f84715d4a1fa2e33311e3e6799dd418e32e2b5d213145016882" exitCode=0 Jul 16 11:05:37.416072 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.415771 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c76d5874-lsbs4" Jul 16 11:05:37.416072 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.415788 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c76d5874-lsbs4" event={"ID":"b3f323d8-8779-4dcf-9e15-e1a55b70317d","Type":"ContainerDied","Data":"c476993975558f84715d4a1fa2e33311e3e6799dd418e32e2b5d213145016882"} Jul 16 11:05:37.416072 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.415857 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c76d5874-lsbs4" event={"ID":"b3f323d8-8779-4dcf-9e15-e1a55b70317d","Type":"ContainerDied","Data":"a483e93129d82ca825c78b7cfde2da94f3e3648e8108f0f0e4a1cc9771db7f9a"} Jul 16 11:05:37.416072 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.415880 2580 scope.go:117] "RemoveContainer" containerID="c476993975558f84715d4a1fa2e33311e3e6799dd418e32e2b5d213145016882" Jul 16 11:05:37.424305 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.424288 2580 scope.go:117] "RemoveContainer" containerID="c476993975558f84715d4a1fa2e33311e3e6799dd418e32e2b5d213145016882" Jul 16 11:05:37.424610 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:05:37.424569 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c476993975558f84715d4a1fa2e33311e3e6799dd418e32e2b5d213145016882\": container with ID starting with c476993975558f84715d4a1fa2e33311e3e6799dd418e32e2b5d213145016882 not found: ID does not exist" containerID="c476993975558f84715d4a1fa2e33311e3e6799dd418e32e2b5d213145016882" Jul 16 11:05:37.424666 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.424620 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c476993975558f84715d4a1fa2e33311e3e6799dd418e32e2b5d213145016882"} err="failed to get container status \"c476993975558f84715d4a1fa2e33311e3e6799dd418e32e2b5d213145016882\": rpc error: code = NotFound desc = could not find container \"c476993975558f84715d4a1fa2e33311e3e6799dd418e32e2b5d213145016882\": container with ID starting with c476993975558f84715d4a1fa2e33311e3e6799dd418e32e2b5d213145016882 not found: ID does not exist" Jul 16 11:05:37.439174 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.439148 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c76d5874-lsbs4"] Jul 16 11:05:37.445149 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:37.445127 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c76d5874-lsbs4"] Jul 16 11:05:38.885076 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:38.885044 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3f323d8-8779-4dcf-9e15-e1a55b70317d" path="/var/lib/kubelet/pods/b3f323d8-8779-4dcf-9e15-e1a55b70317d/volumes" Jul 16 11:05:40.574037 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:40.574010 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:05:40.577931 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:05:40.577907 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-76c4dbdb5b-jhthw" Jul 16 11:06:02.570300 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:02.570261 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:06:02.588860 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:02.588835 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:06:03.526876 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:03.526849 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 11:06:20.336108 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.336074 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-74cf4d787c-qd7g7"] Jul 16 11:06:20.336590 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.336396 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f323d8-8779-4dcf-9e15-e1a55b70317d" containerName="console" Jul 16 11:06:20.336590 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.336407 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f323d8-8779-4dcf-9e15-e1a55b70317d" containerName="console" Jul 16 11:06:20.336590 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.336464 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f323d8-8779-4dcf-9e15-e1a55b70317d" containerName="console" Jul 16 11:06:20.338683 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.338663 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.341960 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.341938 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 16 11:06:20.342128 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.342103 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 16 11:06:20.342128 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.342128 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 16 11:06:20.342295 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.342266 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7xfhm\"" Jul 16 11:06:20.342457 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.342444 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 16 11:06:20.343051 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.343037 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 16 11:06:20.348313 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.348295 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 16 11:06:20.365388 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.365364 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74cf4d787c-qd7g7"] Jul 16 11:06:20.468633 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.468603 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rws\" (UniqueName: \"kubernetes.io/projected/b0a7a63e-e016-4f06-a5aa-8104fed5a765-kube-api-access-68rws\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.468788 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.468649 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a7a63e-e016-4f06-a5aa-8104fed5a765-serving-certs-ca-bundle\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.468788 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.468670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a7a63e-e016-4f06-a5aa-8104fed5a765-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.468788 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.468690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b0a7a63e-e016-4f06-a5aa-8104fed5a765-telemeter-client-tls\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.468788 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.468707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b0a7a63e-e016-4f06-a5aa-8104fed5a765-secret-telemeter-client\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.468788 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.468730 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b0a7a63e-e016-4f06-a5aa-8104fed5a765-federate-client-tls\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.468788 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.468776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0a7a63e-e016-4f06-a5aa-8104fed5a765-metrics-client-ca\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.468973 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.468801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0a7a63e-e016-4f06-a5aa-8104fed5a765-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.569218 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.569194 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a7a63e-e016-4f06-a5aa-8104fed5a765-serving-certs-ca-bundle\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.569343 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.569226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a7a63e-e016-4f06-a5aa-8104fed5a765-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.569343 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.569247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b0a7a63e-e016-4f06-a5aa-8104fed5a765-telemeter-client-tls\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.569343 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.569270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b0a7a63e-e016-4f06-a5aa-8104fed5a765-secret-telemeter-client\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.569343 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.569300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b0a7a63e-e016-4f06-a5aa-8104fed5a765-federate-client-tls\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.569343 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.569341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0a7a63e-e016-4f06-a5aa-8104fed5a765-metrics-client-ca\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.569604 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.569365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0a7a63e-e016-4f06-a5aa-8104fed5a765-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.569604 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.569438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rws\" (UniqueName: \"kubernetes.io/projected/b0a7a63e-e016-4f06-a5aa-8104fed5a765-kube-api-access-68rws\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.570189 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.570156 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0a7a63e-e016-4f06-a5aa-8104fed5a765-metrics-client-ca\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.571106 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.571040 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a7a63e-e016-4f06-a5aa-8104fed5a765-serving-certs-ca-bundle\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.571237 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.571142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a7a63e-e016-4f06-a5aa-8104fed5a765-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.572360 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.572326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b0a7a63e-e016-4f06-a5aa-8104fed5a765-telemeter-client-tls\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.572464 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.572326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0a7a63e-e016-4f06-a5aa-8104fed5a765-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.572831 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.572801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b0a7a63e-e016-4f06-a5aa-8104fed5a765-secret-telemeter-client\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.572920 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.572835 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b0a7a63e-e016-4f06-a5aa-8104fed5a765-federate-client-tls\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.584103 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.584082 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rws\" (UniqueName: \"kubernetes.io/projected/b0a7a63e-e016-4f06-a5aa-8104fed5a765-kube-api-access-68rws\") pod \"telemeter-client-74cf4d787c-qd7g7\" (UID: \"b0a7a63e-e016-4f06-a5aa-8104fed5a765\") " pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.648316 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.648219 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" Jul 16 11:06:20.787226 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:20.787199 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74cf4d787c-qd7g7"] Jul 16 11:06:20.790515 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:06:20.790490 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a7a63e_e016_4f06_a5aa_8104fed5a765.slice/crio-68d35302ac14cd88a2809e0f6b1d49f938ef4b6bb63d92d778e011a7fc7675c9 WatchSource:0}: Error finding container 68d35302ac14cd88a2809e0f6b1d49f938ef4b6bb63d92d778e011a7fc7675c9: Status 404 returned error can't find the container with id 68d35302ac14cd88a2809e0f6b1d49f938ef4b6bb63d92d778e011a7fc7675c9 Jul 16 11:06:21.581822 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:21.581792 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" event={"ID":"b0a7a63e-e016-4f06-a5aa-8104fed5a765","Type":"ContainerStarted","Data":"68d35302ac14cd88a2809e0f6b1d49f938ef4b6bb63d92d778e011a7fc7675c9"} Jul 16 11:06:23.593297 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:23.593264 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" event={"ID":"b0a7a63e-e016-4f06-a5aa-8104fed5a765","Type":"ContainerStarted","Data":"5fa55107951cad4ae9d4531f3186f2073a10e94feaab295439a3dc3cec7ccfa4"} Jul 16 11:06:23.593716 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:23.593322 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" event={"ID":"b0a7a63e-e016-4f06-a5aa-8104fed5a765","Type":"ContainerStarted","Data":"5af68f75c4b17a4c4338a9336f330f04038f4769e9382ada6037861e9b2165be"} Jul 16 11:06:23.593716 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:23.593342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" event={"ID":"b0a7a63e-e016-4f06-a5aa-8104fed5a765","Type":"ContainerStarted","Data":"0972e5ad533e7e1efb10f5380a97f8b14f0429d0868b6ff2f1f9df98bf9019dd"} Jul 16 11:06:23.616880 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:23.616831 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-74cf4d787c-qd7g7" podStartSLOduration=1.82099637 podStartE2EDuration="3.616809864s" totalImagesPullingTime="1.795813494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:06:23.614062187 +0000 UTC m=+155.447410869" watchObservedRunningTime="2026-07-16 11:06:23.616809864 +0000 UTC m=+155.450158543" Jul 16 11:06:44.430930 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:06:44.430853 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bc8dc6588-746qt"] Jul 16 11:07:09.455635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.455592 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bc8dc6588-746qt" podUID="e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7" containerName="console" containerID="cri-o://a9b9050d4ed44d04d0c1483634ed2499e7edec75a4ac851cf696d8b053d43df3" gracePeriod=15 Jul 16 11:07:09.711763 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.711705 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:07:09.777637 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.777594 2580 generic.go:356] "Generic (PLEG): container finished" podID="e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7" containerID="a9b9050d4ed44d04d0c1483634ed2499e7edec75a4ac851cf696d8b053d43df3" exitCode=0 Jul 16 11:07:09.777810 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.777685 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc8dc6588-746qt" Jul 16 11:07:09.777810 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.777707 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc8dc6588-746qt" event={"ID":"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7","Type":"ContainerDied","Data":"a9b9050d4ed44d04d0c1483634ed2499e7edec75a4ac851cf696d8b053d43df3"} Jul 16 11:07:09.777810 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.777752 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc8dc6588-746qt" event={"ID":"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7","Type":"ContainerDied","Data":"28da74c5d177e9a790da56713d7df7fe7f690b7d3c48477937f941576c88014a"} Jul 16 11:07:09.777810 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.777776 2580 scope.go:117] "RemoveContainer" containerID="a9b9050d4ed44d04d0c1483634ed2499e7edec75a4ac851cf696d8b053d43df3" Jul 16 11:07:09.785605 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.785568 2580 scope.go:117] "RemoveContainer" containerID="a9b9050d4ed44d04d0c1483634ed2499e7edec75a4ac851cf696d8b053d43df3" Jul 16 11:07:09.785872 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:07:09.785854 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b9050d4ed44d04d0c1483634ed2499e7edec75a4ac851cf696d8b053d43df3\": container with ID starting with a9b9050d4ed44d04d0c1483634ed2499e7edec75a4ac851cf696d8b053d43df3 not found: ID does not exist" containerID="a9b9050d4ed44d04d0c1483634ed2499e7edec75a4ac851cf696d8b053d43df3" Jul 16 11:07:09.785921 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.785883 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b9050d4ed44d04d0c1483634ed2499e7edec75a4ac851cf696d8b053d43df3"} err="failed to get container status \"a9b9050d4ed44d04d0c1483634ed2499e7edec75a4ac851cf696d8b053d43df3\": rpc error: code = NotFound desc = could not find container \"a9b9050d4ed44d04d0c1483634ed2499e7edec75a4ac851cf696d8b053d43df3\": container with ID starting with a9b9050d4ed44d04d0c1483634ed2499e7edec75a4ac851cf696d8b053d43df3 not found: ID does not exist" Jul 16 11:07:09.795142 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.795119 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-service-ca\") pod \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " Jul 16 11:07:09.795252 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.795154 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-serving-cert\") pod \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " Jul 16 11:07:09.795252 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.795177 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb6lp\" (UniqueName: \"kubernetes.io/projected/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-kube-api-access-zb6lp\") pod \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " Jul 16 11:07:09.795252 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.795214 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-oauth-config\") pod \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " Jul 16 11:07:09.795407 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.795308 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-oauth-serving-cert\") pod \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " Jul 16 11:07:09.795407 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.795352 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-config\") pod \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " Jul 16 11:07:09.795407 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.795396 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-trusted-ca-bundle\") pod \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\" (UID: \"e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7\") " Jul 16 11:07:09.795551 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.795494 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-service-ca" (OuterVolumeSpecName: "service-ca") pod "e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7" (UID: "e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:07:09.795715 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.795696 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-service-ca\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:07:09.795855 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.795727 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7" (UID: "e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:07:09.795956 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.795736 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-config" (OuterVolumeSpecName: "console-config") pod "e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7" (UID: "e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:07:09.795956 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.795926 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7" (UID: "e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:07:09.797476 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.797449 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-kube-api-access-zb6lp" (OuterVolumeSpecName: "kube-api-access-zb6lp") pod "e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7" (UID: "e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7"). InnerVolumeSpecName "kube-api-access-zb6lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:07:09.797757 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.797735 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7" (UID: "e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:07:09.797827 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.797809 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7" (UID: "e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:07:09.896445 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.896396 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-trusted-ca-bundle\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:07:09.896445 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.896438 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-serving-cert\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:07:09.896445 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.896450 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb6lp\" (UniqueName: \"kubernetes.io/projected/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-kube-api-access-zb6lp\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:07:09.896445 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.896459 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-oauth-config\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:07:09.896741 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.896470 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-oauth-serving-cert\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:07:09.896741 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:09.896478 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7-console-config\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:07:10.100955 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:10.100921 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bc8dc6588-746qt"] Jul 16 11:07:10.106150 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:10.106126 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bc8dc6588-746qt"] Jul 16 11:07:10.885230 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:10.885198 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7" path="/var/lib/kubelet/pods/e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7/volumes" Jul 16 11:07:36.863923 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:36.863890 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2"] Jul 16 11:07:36.864371 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:36.864198 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7" containerName="console" Jul 16 11:07:36.864371 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:36.864208 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7" containerName="console" Jul 16 11:07:36.864371 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:36.864260 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e85bc60c-62f2-4ea5-9f38-bcf1366f3bf7" containerName="console" Jul 16 11:07:36.868927 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:36.868903 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" Jul 16 11:07:36.872622 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:36.872601 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 16 11:07:36.873055 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:36.873041 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 16 11:07:36.873118 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:36.873057 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 16 11:07:36.873672 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:36.873657 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 16 11:07:36.884601 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:36.884561 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2"] Jul 16 11:07:36.917643 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:36.917617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wj5n\" (UniqueName: \"kubernetes.io/projected/5abcc481-f04b-41f8-a702-aef4c7115349-kube-api-access-8wj5n\") pod \"klusterlet-addon-workmgr-6d8d56f7cb-lkwd2\" (UID: \"5abcc481-f04b-41f8-a702-aef4c7115349\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" Jul 16 11:07:36.917773 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:36.917689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5abcc481-f04b-41f8-a702-aef4c7115349-tmp\") pod \"klusterlet-addon-workmgr-6d8d56f7cb-lkwd2\" (UID: \"5abcc481-f04b-41f8-a702-aef4c7115349\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" Jul 16 11:07:36.917773 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:36.917712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5abcc481-f04b-41f8-a702-aef4c7115349-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d8d56f7cb-lkwd2\" (UID: \"5abcc481-f04b-41f8-a702-aef4c7115349\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" Jul 16 11:07:37.018480 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:37.018451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5abcc481-f04b-41f8-a702-aef4c7115349-tmp\") pod \"klusterlet-addon-workmgr-6d8d56f7cb-lkwd2\" (UID: \"5abcc481-f04b-41f8-a702-aef4c7115349\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" Jul 16 11:07:37.018693 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:37.018497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5abcc481-f04b-41f8-a702-aef4c7115349-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d8d56f7cb-lkwd2\" (UID: \"5abcc481-f04b-41f8-a702-aef4c7115349\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" Jul 16 11:07:37.018766 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:37.018733 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wj5n\" (UniqueName: \"kubernetes.io/projected/5abcc481-f04b-41f8-a702-aef4c7115349-kube-api-access-8wj5n\") pod \"klusterlet-addon-workmgr-6d8d56f7cb-lkwd2\" (UID: \"5abcc481-f04b-41f8-a702-aef4c7115349\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" Jul 16 11:07:37.018885 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:37.018863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5abcc481-f04b-41f8-a702-aef4c7115349-tmp\") pod \"klusterlet-addon-workmgr-6d8d56f7cb-lkwd2\" (UID: \"5abcc481-f04b-41f8-a702-aef4c7115349\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" Jul 16 11:07:37.020975 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:37.020951 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5abcc481-f04b-41f8-a702-aef4c7115349-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d8d56f7cb-lkwd2\" (UID: \"5abcc481-f04b-41f8-a702-aef4c7115349\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" Jul 16 11:07:37.026953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:37.026934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wj5n\" (UniqueName: \"kubernetes.io/projected/5abcc481-f04b-41f8-a702-aef4c7115349-kube-api-access-8wj5n\") pod \"klusterlet-addon-workmgr-6d8d56f7cb-lkwd2\" (UID: \"5abcc481-f04b-41f8-a702-aef4c7115349\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" Jul 16 11:07:37.178209 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:37.178126 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" Jul 16 11:07:37.305270 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:37.305177 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2"] Jul 16 11:07:37.308148 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:07:37.308113 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abcc481_f04b_41f8_a702_aef4c7115349.slice/crio-b9d08b52bd53d72da51dccaa5644fbff1bacc0b9c9f50321cea88ae42f57c139 WatchSource:0}: Error finding container b9d08b52bd53d72da51dccaa5644fbff1bacc0b9c9f50321cea88ae42f57c139: Status 404 returned error can't find the container with id b9d08b52bd53d72da51dccaa5644fbff1bacc0b9c9f50321cea88ae42f57c139 Jul 16 11:07:37.884120 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:37.884085 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" event={"ID":"5abcc481-f04b-41f8-a702-aef4c7115349","Type":"ContainerStarted","Data":"b9d08b52bd53d72da51dccaa5644fbff1bacc0b9c9f50321cea88ae42f57c139"} Jul 16 11:07:44.912478 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:44.912393 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" event={"ID":"5abcc481-f04b-41f8-a702-aef4c7115349","Type":"ContainerStarted","Data":"02ac7b73aef3dce7ff3089ba3de99a8be1d4d477feebf2b4ee20e04a9cfaa476"} Jul 16 11:07:44.912860 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:44.912604 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" Jul 16 11:07:44.914264 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:44.914242 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" Jul 16 11:07:44.930861 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:07:44.930819 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8d56f7cb-lkwd2" podStartSLOduration=1.647027807 podStartE2EDuration="8.930804682s" totalImagesPullingTime="7.283776875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:07:44.929588533 +0000 UTC m=+236.762937213" watchObservedRunningTime="2026-07-16 11:07:44.930804682 +0000 UTC m=+236.764153369" Jul 16 11:08:14.525735 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:14.525658 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bw2fz"] Jul 16 11:08:14.527777 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:14.527760 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bw2fz" Jul 16 11:08:14.530855 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:14.530838 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 11:08:14.531072 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:14.531059 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6k6vf\"" Jul 16 11:08:14.531807 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:14.531787 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 11:08:14.537511 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:14.537491 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bw2fz"] Jul 16 11:08:14.636152 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:14.636117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8633bc8b-95ff-46fc-b44e-5b101d93996f-bound-sa-token\") pod \"cert-manager-55b4d954d5-bw2fz\" (UID: \"8633bc8b-95ff-46fc-b44e-5b101d93996f\") " pod="cert-manager/cert-manager-55b4d954d5-bw2fz" Jul 16 11:08:14.636358 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:14.636229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4w4m\" (UniqueName: \"kubernetes.io/projected/8633bc8b-95ff-46fc-b44e-5b101d93996f-kube-api-access-f4w4m\") pod \"cert-manager-55b4d954d5-bw2fz\" (UID: \"8633bc8b-95ff-46fc-b44e-5b101d93996f\") " pod="cert-manager/cert-manager-55b4d954d5-bw2fz" Jul 16 11:08:14.737490 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:14.737451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4w4m\" (UniqueName: \"kubernetes.io/projected/8633bc8b-95ff-46fc-b44e-5b101d93996f-kube-api-access-f4w4m\") pod \"cert-manager-55b4d954d5-bw2fz\" (UID: \"8633bc8b-95ff-46fc-b44e-5b101d93996f\") " pod="cert-manager/cert-manager-55b4d954d5-bw2fz" Jul 16 11:08:14.737667 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:14.737511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8633bc8b-95ff-46fc-b44e-5b101d93996f-bound-sa-token\") pod \"cert-manager-55b4d954d5-bw2fz\" (UID: \"8633bc8b-95ff-46fc-b44e-5b101d93996f\") " pod="cert-manager/cert-manager-55b4d954d5-bw2fz" Jul 16 11:08:14.746902 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:14.746865 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8633bc8b-95ff-46fc-b44e-5b101d93996f-bound-sa-token\") pod \"cert-manager-55b4d954d5-bw2fz\" (UID: \"8633bc8b-95ff-46fc-b44e-5b101d93996f\") " pod="cert-manager/cert-manager-55b4d954d5-bw2fz" Jul 16 11:08:14.747149 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:14.747129 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4w4m\" (UniqueName: \"kubernetes.io/projected/8633bc8b-95ff-46fc-b44e-5b101d93996f-kube-api-access-f4w4m\") pod \"cert-manager-55b4d954d5-bw2fz\" (UID: \"8633bc8b-95ff-46fc-b44e-5b101d93996f\") " pod="cert-manager/cert-manager-55b4d954d5-bw2fz" Jul 16 11:08:14.852545 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:14.852456 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bw2fz" Jul 16 11:08:14.978208 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:14.978178 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bw2fz"] Jul 16 11:08:14.981522 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:08:14.981498 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8633bc8b_95ff_46fc_b44e_5b101d93996f.slice/crio-fb59b6434f008509168a2dfa18952265ef3c11ca7d93179990eec741e6b5ee46 WatchSource:0}: Error finding container fb59b6434f008509168a2dfa18952265ef3c11ca7d93179990eec741e6b5ee46: Status 404 returned error can't find the container with id fb59b6434f008509168a2dfa18952265ef3c11ca7d93179990eec741e6b5ee46 Jul 16 11:08:15.033131 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:15.033097 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bw2fz" event={"ID":"8633bc8b-95ff-46fc-b44e-5b101d93996f","Type":"ContainerStarted","Data":"fb59b6434f008509168a2dfa18952265ef3c11ca7d93179990eec741e6b5ee46"} Jul 16 11:08:19.051271 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.051230 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bw2fz" event={"ID":"8633bc8b-95ff-46fc-b44e-5b101d93996f","Type":"ContainerStarted","Data":"186209c8ee11cd2dd6f1e733502289277948d22c1f8079fe87e71e9e344a1c1a"} Jul 16 11:08:19.074279 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.074233 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bw2fz" podStartSLOduration=1.972108628 podStartE2EDuration="5.07421687s" totalImagesPullingTime="3.102108242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:08:19.073246035 +0000 UTC m=+270.906594719" watchObservedRunningTime="2026-07-16 11:08:19.07421687 +0000 UTC m=+270.907565551" Jul 16 11:08:19.548985 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.548951 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9m7wv"] Jul 16 11:08:19.551221 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.551202 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9m7wv" Jul 16 11:08:19.554276 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.554256 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 11:08:19.554603 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.554562 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pk2m5\"" Jul 16 11:08:19.555488 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.555469 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 11:08:19.566077 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.566056 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9m7wv"] Jul 16 11:08:19.683365 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.683323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kcl7\" (UniqueName: \"kubernetes.io/projected/3826c964-b173-4bdb-a952-604a9631f721-kube-api-access-5kcl7\") pod \"openshift-lws-operator-fd8ccff4c-9m7wv\" (UID: \"3826c964-b173-4bdb-a952-604a9631f721\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9m7wv" Jul 16 11:08:19.683551 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.683392 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3826c964-b173-4bdb-a952-604a9631f721-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9m7wv\" (UID: \"3826c964-b173-4bdb-a952-604a9631f721\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9m7wv" Jul 16 11:08:19.784268 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.784235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcl7\" (UniqueName: \"kubernetes.io/projected/3826c964-b173-4bdb-a952-604a9631f721-kube-api-access-5kcl7\") pod \"openshift-lws-operator-fd8ccff4c-9m7wv\" (UID: \"3826c964-b173-4bdb-a952-604a9631f721\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9m7wv" Jul 16 11:08:19.784428 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.784290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3826c964-b173-4bdb-a952-604a9631f721-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9m7wv\" (UID: \"3826c964-b173-4bdb-a952-604a9631f721\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9m7wv" Jul 16 11:08:19.784739 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.784722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3826c964-b173-4bdb-a952-604a9631f721-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9m7wv\" (UID: \"3826c964-b173-4bdb-a952-604a9631f721\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9m7wv" Jul 16 11:08:19.799665 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.799601 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcl7\" (UniqueName: \"kubernetes.io/projected/3826c964-b173-4bdb-a952-604a9631f721-kube-api-access-5kcl7\") pod \"openshift-lws-operator-fd8ccff4c-9m7wv\" (UID: \"3826c964-b173-4bdb-a952-604a9631f721\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9m7wv" Jul 16 11:08:19.860956 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.860924 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9m7wv" Jul 16 11:08:19.988476 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:19.988332 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9m7wv"] Jul 16 11:08:19.991063 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:08:19.991036 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3826c964_b173_4bdb_a952_604a9631f721.slice/crio-8fbb7b747d5ca5e091f5d3ae0bac86440442d47fff641f0cdf54209771b4da78 WatchSource:0}: Error finding container 8fbb7b747d5ca5e091f5d3ae0bac86440442d47fff641f0cdf54209771b4da78: Status 404 returned error can't find the container with id 8fbb7b747d5ca5e091f5d3ae0bac86440442d47fff641f0cdf54209771b4da78 Jul 16 11:08:20.060755 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:20.060677 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9m7wv" event={"ID":"3826c964-b173-4bdb-a952-604a9631f721","Type":"ContainerStarted","Data":"8fbb7b747d5ca5e091f5d3ae0bac86440442d47fff641f0cdf54209771b4da78"} Jul 16 11:08:23.076661 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:23.076626 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9m7wv" event={"ID":"3826c964-b173-4bdb-a952-604a9631f721","Type":"ContainerStarted","Data":"0a80e6d05a0004a66e6e319e5da2586c555c30dc33b735d0b08029f6f5553cdb"} Jul 16 11:08:23.108449 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:23.108394 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9m7wv" podStartSLOduration=1.708666856 podStartE2EDuration="4.108379051s" totalImagesPullingTime="2.399712195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:08:23.10794339 +0000 UTC m=+274.941292092" watchObservedRunningTime="2026-07-16 11:08:23.108379051 +0000 UTC m=+274.941727734" Jul 16 11:08:40.485909 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.485876 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48"] Jul 16 11:08:40.488857 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.488841 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:40.493313 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.493291 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mdb7c\"" Jul 16 11:08:40.494556 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.494525 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 11:08:40.494790 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.494775 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 11:08:40.496492 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.496473 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 11:08:40.516625 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.516595 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48"] Jul 16 11:08:40.540245 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.540221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b276feaa-ccf3-4f4c-8e28-7523b65e19b1-metrics-cert\") pod \"lws-controller-manager-5dbfbcbfc5-zkb48\" (UID: \"b276feaa-ccf3-4f4c-8e28-7523b65e19b1\") " pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:40.540367 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.540269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b276feaa-ccf3-4f4c-8e28-7523b65e19b1-manager-config\") pod \"lws-controller-manager-5dbfbcbfc5-zkb48\" (UID: \"b276feaa-ccf3-4f4c-8e28-7523b65e19b1\") " pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:40.540367 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.540303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b276feaa-ccf3-4f4c-8e28-7523b65e19b1-cert\") pod \"lws-controller-manager-5dbfbcbfc5-zkb48\" (UID: \"b276feaa-ccf3-4f4c-8e28-7523b65e19b1\") " pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:40.540367 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.540329 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qv66\" (UniqueName: \"kubernetes.io/projected/b276feaa-ccf3-4f4c-8e28-7523b65e19b1-kube-api-access-6qv66\") pod \"lws-controller-manager-5dbfbcbfc5-zkb48\" (UID: \"b276feaa-ccf3-4f4c-8e28-7523b65e19b1\") " pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:40.641340 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.641302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b276feaa-ccf3-4f4c-8e28-7523b65e19b1-manager-config\") pod \"lws-controller-manager-5dbfbcbfc5-zkb48\" (UID: \"b276feaa-ccf3-4f4c-8e28-7523b65e19b1\") " pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:40.641534 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.641356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b276feaa-ccf3-4f4c-8e28-7523b65e19b1-cert\") pod \"lws-controller-manager-5dbfbcbfc5-zkb48\" (UID: \"b276feaa-ccf3-4f4c-8e28-7523b65e19b1\") " pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:40.641534 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.641377 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qv66\" (UniqueName: \"kubernetes.io/projected/b276feaa-ccf3-4f4c-8e28-7523b65e19b1-kube-api-access-6qv66\") pod \"lws-controller-manager-5dbfbcbfc5-zkb48\" (UID: \"b276feaa-ccf3-4f4c-8e28-7523b65e19b1\") " pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:40.641699 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.641550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b276feaa-ccf3-4f4c-8e28-7523b65e19b1-metrics-cert\") pod \"lws-controller-manager-5dbfbcbfc5-zkb48\" (UID: \"b276feaa-ccf3-4f4c-8e28-7523b65e19b1\") " pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:40.642027 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.642001 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b276feaa-ccf3-4f4c-8e28-7523b65e19b1-manager-config\") pod \"lws-controller-manager-5dbfbcbfc5-zkb48\" (UID: \"b276feaa-ccf3-4f4c-8e28-7523b65e19b1\") " pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:40.643868 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.643840 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b276feaa-ccf3-4f4c-8e28-7523b65e19b1-metrics-cert\") pod \"lws-controller-manager-5dbfbcbfc5-zkb48\" (UID: \"b276feaa-ccf3-4f4c-8e28-7523b65e19b1\") " pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:40.643957 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.643885 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b276feaa-ccf3-4f4c-8e28-7523b65e19b1-cert\") pod \"lws-controller-manager-5dbfbcbfc5-zkb48\" (UID: \"b276feaa-ccf3-4f4c-8e28-7523b65e19b1\") " pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:40.653008 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.652983 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qv66\" (UniqueName: \"kubernetes.io/projected/b276feaa-ccf3-4f4c-8e28-7523b65e19b1-kube-api-access-6qv66\") pod \"lws-controller-manager-5dbfbcbfc5-zkb48\" (UID: \"b276feaa-ccf3-4f4c-8e28-7523b65e19b1\") " pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:40.798531 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.798500 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:40.944276 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:40.944248 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48"] Jul 16 11:08:40.946688 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:08:40.946658 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb276feaa_ccf3_4f4c_8e28_7523b65e19b1.slice/crio-05fd21b49c62a9b0f31327b11587a916e952622a6f4864e616cce0a46b0f1938 WatchSource:0}: Error finding container 05fd21b49c62a9b0f31327b11587a916e952622a6f4864e616cce0a46b0f1938: Status 404 returned error can't find the container with id 05fd21b49c62a9b0f31327b11587a916e952622a6f4864e616cce0a46b0f1938 Jul 16 11:08:41.147036 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:41.146954 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" event={"ID":"b276feaa-ccf3-4f4c-8e28-7523b65e19b1","Type":"ContainerStarted","Data":"05fd21b49c62a9b0f31327b11587a916e952622a6f4864e616cce0a46b0f1938"} Jul 16 11:08:42.682304 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.682274 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7"] Jul 16 11:08:42.685989 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.685971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.688738 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.688715 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 11:08:42.688890 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.688810 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zqz4c\"" Jul 16 11:08:42.688890 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.688831 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 11:08:42.689208 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.689189 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 11:08:42.689556 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.689536 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 11:08:42.701761 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.701738 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7"] Jul 16 11:08:42.758284 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.758252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5d927bf-89d0-477c-8753-5f1a218e9c29-tmp\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.758458 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.758295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5d927bf-89d0-477c-8753-5f1a218e9c29-webhook-cert\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.758458 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.758341 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b5d927bf-89d0-477c-8753-5f1a218e9c29-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.758458 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.758398 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b5d927bf-89d0-477c-8753-5f1a218e9c29-manifests\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.758458 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.758420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-645pw\" (UniqueName: \"kubernetes.io/projected/b5d927bf-89d0-477c-8753-5f1a218e9c29-kube-api-access-645pw\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.859161 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.859120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b5d927bf-89d0-477c-8753-5f1a218e9c29-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.859360 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.859299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b5d927bf-89d0-477c-8753-5f1a218e9c29-manifests\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.859360 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.859354 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-645pw\" (UniqueName: \"kubernetes.io/projected/b5d927bf-89d0-477c-8753-5f1a218e9c29-kube-api-access-645pw\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.859478 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.859425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5d927bf-89d0-477c-8753-5f1a218e9c29-tmp\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.859529 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.859479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5d927bf-89d0-477c-8753-5f1a218e9c29-webhook-cert\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.859845 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.859808 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b5d927bf-89d0-477c-8753-5f1a218e9c29-manifests\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.862276 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.862212 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5d927bf-89d0-477c-8753-5f1a218e9c29-tmp\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.862474 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.862429 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5d927bf-89d0-477c-8753-5f1a218e9c29-webhook-cert\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.862704 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.862681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b5d927bf-89d0-477c-8753-5f1a218e9c29-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.871544 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.871507 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-645pw\" (UniqueName: \"kubernetes.io/projected/b5d927bf-89d0-477c-8753-5f1a218e9c29-kube-api-access-645pw\") pod \"opendatahub-operator-controller-manager-58ffff6c5d-lxpk7\" (UID: \"b5d927bf-89d0-477c-8753-5f1a218e9c29\") " pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:42.999304 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:42.999267 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:43.151058 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:43.151031 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7"] Jul 16 11:08:43.153458 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:08:43.153429 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d927bf_89d0_477c_8753_5f1a218e9c29.slice/crio-4f4e7b2cfe4bb5f2fc8a65315a44d29f1968d9210a18c2e0d6496e2bf63c028b WatchSource:0}: Error finding container 4f4e7b2cfe4bb5f2fc8a65315a44d29f1968d9210a18c2e0d6496e2bf63c028b: Status 404 returned error can't find the container with id 4f4e7b2cfe4bb5f2fc8a65315a44d29f1968d9210a18c2e0d6496e2bf63c028b Jul 16 11:08:43.156919 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:43.156884 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" event={"ID":"b276feaa-ccf3-4f4c-8e28-7523b65e19b1","Type":"ContainerStarted","Data":"1ca2b7339d3c23a12601cf68d73b40bc9bb8775aff5aa5e8338dafdfc3233b90"} Jul 16 11:08:43.157036 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:43.156949 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:08:43.185185 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:43.185143 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" podStartSLOduration=1.175950299 podStartE2EDuration="3.18512742s" totalImagesPullingTime="2.009177121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:08:43.183442688 +0000 UTC m=+295.016791370" watchObservedRunningTime="2026-07-16 11:08:43.18512742 +0000 UTC m=+295.018476111" Jul 16 11:08:44.163125 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:44.163092 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" event={"ID":"b5d927bf-89d0-477c-8753-5f1a218e9c29","Type":"ContainerStarted","Data":"4f4e7b2cfe4bb5f2fc8a65315a44d29f1968d9210a18c2e0d6496e2bf63c028b"} Jul 16 11:08:46.173027 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:46.172994 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" event={"ID":"b5d927bf-89d0-477c-8753-5f1a218e9c29","Type":"ContainerStarted","Data":"e0ae338b4435471d91d7c203ff3d0a8d05148469523f4feb1d57ce3dff6203d3"} Jul 16 11:08:48.182973 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:48.182941 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5d927bf-89d0-477c-8753-5f1a218e9c29" containerID="e0ae338b4435471d91d7c203ff3d0a8d05148469523f4feb1d57ce3dff6203d3" exitCode=0 Jul 16 11:08:48.183349 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:48.183014 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" event={"ID":"b5d927bf-89d0-477c-8753-5f1a218e9c29","Type":"ContainerDied","Data":"e0ae338b4435471d91d7c203ff3d0a8d05148469523f4feb1d57ce3dff6203d3"} Jul 16 11:08:48.586499 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:48.585957 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:08:48.589777 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:48.589752 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:08:48.593550 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:48.593526 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 11:08:49.189819 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:49.189782 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" event={"ID":"b5d927bf-89d0-477c-8753-5f1a218e9c29","Type":"ContainerStarted","Data":"fe934dd557082f8821110eb6db99e1344aa49be54bb9f612f77467f35533a735"} Jul 16 11:08:49.192029 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:49.190012 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:08:49.216874 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:49.216823 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" podStartSLOduration=2.864799262 podStartE2EDuration="7.216807066s" totalImagesPullingTime="3.263007666s" totalInitContainerRuntime="1.089000138s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:08:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:08:49.214723435 +0000 UTC m=+301.048072118" watchObservedRunningTime="2026-07-16 11:08:49.216807066 +0000 UTC m=+301.050155759" Jul 16 11:08:54.166052 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:08:54.166024 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5dbfbcbfc5-zkb48" Jul 16 11:09:00.196588 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:00.196544 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58ffff6c5d-lxpk7" Jul 16 11:09:02.604286 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.604257 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866"] Jul 16 11:09:02.613463 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.613439 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866" Jul 16 11:09:02.616804 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.616746 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 11:09:02.617053 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.617034 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 11:09:02.618184 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.618015 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-f4mtd\"" Jul 16 11:09:02.618184 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.618047 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 11:09:02.618184 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.618049 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 11:09:02.620797 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.620765 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866"] Jul 16 11:09:02.753465 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.753426 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f021cc-9bcf-41a7-8174-661b76023a27-tls-certs\") pod \"kube-auth-proxy-b4b6f7dfc-zb866\" (UID: \"77f021cc-9bcf-41a7-8174-661b76023a27\") " pod="openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866" Jul 16 11:09:02.753465 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.753464 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77f021cc-9bcf-41a7-8174-661b76023a27-tmp\") pod \"kube-auth-proxy-b4b6f7dfc-zb866\" (UID: \"77f021cc-9bcf-41a7-8174-661b76023a27\") " pod="openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866" Jul 16 11:09:02.753785 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.753501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpb6q\" (UniqueName: \"kubernetes.io/projected/77f021cc-9bcf-41a7-8174-661b76023a27-kube-api-access-qpb6q\") pod \"kube-auth-proxy-b4b6f7dfc-zb866\" (UID: \"77f021cc-9bcf-41a7-8174-661b76023a27\") " pod="openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866" Jul 16 11:09:02.854692 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.854563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f021cc-9bcf-41a7-8174-661b76023a27-tls-certs\") pod \"kube-auth-proxy-b4b6f7dfc-zb866\" (UID: \"77f021cc-9bcf-41a7-8174-661b76023a27\") " pod="openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866" Jul 16 11:09:02.854692 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.854644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77f021cc-9bcf-41a7-8174-661b76023a27-tmp\") pod \"kube-auth-proxy-b4b6f7dfc-zb866\" (UID: \"77f021cc-9bcf-41a7-8174-661b76023a27\") " pod="openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866" Jul 16 11:09:02.854692 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.854680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpb6q\" (UniqueName: \"kubernetes.io/projected/77f021cc-9bcf-41a7-8174-661b76023a27-kube-api-access-qpb6q\") pod \"kube-auth-proxy-b4b6f7dfc-zb866\" (UID: \"77f021cc-9bcf-41a7-8174-661b76023a27\") " pod="openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866" Jul 16 11:09:02.856986 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.856955 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77f021cc-9bcf-41a7-8174-661b76023a27-tmp\") pod \"kube-auth-proxy-b4b6f7dfc-zb866\" (UID: \"77f021cc-9bcf-41a7-8174-661b76023a27\") " pod="openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866" Jul 16 11:09:02.857105 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.857077 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f021cc-9bcf-41a7-8174-661b76023a27-tls-certs\") pod \"kube-auth-proxy-b4b6f7dfc-zb866\" (UID: \"77f021cc-9bcf-41a7-8174-661b76023a27\") " pod="openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866" Jul 16 11:09:02.867523 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.867491 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpb6q\" (UniqueName: \"kubernetes.io/projected/77f021cc-9bcf-41a7-8174-661b76023a27-kube-api-access-qpb6q\") pod \"kube-auth-proxy-b4b6f7dfc-zb866\" (UID: \"77f021cc-9bcf-41a7-8174-661b76023a27\") " pod="openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866" Jul 16 11:09:02.924932 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:02.924896 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866" Jul 16 11:09:03.058386 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:03.058234 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866"] Jul 16 11:09:03.061043 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:09:03.061017 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f021cc_9bcf_41a7_8174_661b76023a27.slice/crio-498805c99686e7894817a93d964478d6febd55f8384203b0a5dbfb717fc7b9c4 WatchSource:0}: Error finding container 498805c99686e7894817a93d964478d6febd55f8384203b0a5dbfb717fc7b9c4: Status 404 returned error can't find the container with id 498805c99686e7894817a93d964478d6febd55f8384203b0a5dbfb717fc7b9c4 Jul 16 11:09:03.062836 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:03.062819 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 11:09:03.254498 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:03.254456 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866" event={"ID":"77f021cc-9bcf-41a7-8174-661b76023a27","Type":"ContainerStarted","Data":"498805c99686e7894817a93d964478d6febd55f8384203b0a5dbfb717fc7b9c4"} Jul 16 11:09:07.274406 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:07.274367 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866" event={"ID":"77f021cc-9bcf-41a7-8174-661b76023a27","Type":"ContainerStarted","Data":"b1a4a8d52c9588c212ab3d6d627f5cc3687ba9d778fc8290a7f9ca6394690ffe"} Jul 16 11:09:07.292880 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:09:07.292824 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b4b6f7dfc-zb866" podStartSLOduration=2.1395279560000002 podStartE2EDuration="5.292803671s" totalImagesPullingTime="3.153275715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:09:07.292213476 +0000 UTC m=+319.125562158" watchObservedRunningTime="2026-07-16 11:09:07.292803671 +0000 UTC m=+319.126152356" Jul 16 11:10:54.835035 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:54.835004 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x768f"] Jul 16 11:10:54.838274 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:54.838255 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x768f" Jul 16 11:10:54.840699 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:54.840675 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 11:10:54.841752 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:54.841735 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2ksxh\"" Jul 16 11:10:54.841752 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:54.841746 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 11:10:54.856286 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:54.856261 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x768f"] Jul 16 11:10:54.968743 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:54.968711 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl7pl\" (UniqueName: \"kubernetes.io/projected/68d3de97-ebf5-4eef-9dd1-85bfb37060e3-kube-api-access-dl7pl\") pod \"limitador-operator-controller-manager-859d7b78bd-x768f\" (UID: \"68d3de97-ebf5-4eef-9dd1-85bfb37060e3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x768f" Jul 16 11:10:55.069417 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:55.069369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl7pl\" (UniqueName: \"kubernetes.io/projected/68d3de97-ebf5-4eef-9dd1-85bfb37060e3-kube-api-access-dl7pl\") pod \"limitador-operator-controller-manager-859d7b78bd-x768f\" (UID: \"68d3de97-ebf5-4eef-9dd1-85bfb37060e3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x768f" Jul 16 11:10:55.078836 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:55.078813 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl7pl\" (UniqueName: \"kubernetes.io/projected/68d3de97-ebf5-4eef-9dd1-85bfb37060e3-kube-api-access-dl7pl\") pod \"limitador-operator-controller-manager-859d7b78bd-x768f\" (UID: \"68d3de97-ebf5-4eef-9dd1-85bfb37060e3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x768f" Jul 16 11:10:55.149084 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:55.149000 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x768f" Jul 16 11:10:55.282759 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:55.282735 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x768f"] Jul 16 11:10:55.285256 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:10:55.285219 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d3de97_ebf5_4eef_9dd1_85bfb37060e3.slice/crio-9989b04349ee0eabd1393a48d5cbd18803d2fe9a3d0e299d84b21c4d3186ba57 WatchSource:0}: Error finding container 9989b04349ee0eabd1393a48d5cbd18803d2fe9a3d0e299d84b21c4d3186ba57: Status 404 returned error can't find the container with id 9989b04349ee0eabd1393a48d5cbd18803d2fe9a3d0e299d84b21c4d3186ba57 Jul 16 11:10:55.748456 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:55.748425 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x768f" event={"ID":"68d3de97-ebf5-4eef-9dd1-85bfb37060e3","Type":"ContainerStarted","Data":"9989b04349ee0eabd1393a48d5cbd18803d2fe9a3d0e299d84b21c4d3186ba57"} Jul 16 11:10:57.760141 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:57.760098 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x768f" event={"ID":"68d3de97-ebf5-4eef-9dd1-85bfb37060e3","Type":"ContainerStarted","Data":"2beb40427b38192d57e9e35e4518adf8315938d2eb4fdd7db7ccd737908ccd04"} Jul 16 11:10:57.760538 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:57.760229 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x768f" Jul 16 11:10:57.790811 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:10:57.790767 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x768f" podStartSLOduration=2.089869148 podStartE2EDuration="3.790752039s" totalImagesPullingTime="1.700882891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:10:57.790101961 +0000 UTC m=+429.623450645" watchObservedRunningTime="2026-07-16 11:10:57.790752039 +0000 UTC m=+429.624100730" Jul 16 11:11:08.766743 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:08.766711 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x768f" Jul 16 11:11:27.297980 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.297943 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84cf84f4b5-gbjzw"] Jul 16 11:11:27.302020 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.301998 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.304797 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.304774 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 11:11:27.304931 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.304887 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 11:11:27.306030 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.305971 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 11:11:27.306030 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.306005 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 11:11:27.306030 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.306009 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bst9c\"" Jul 16 11:11:27.306275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.306041 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 11:11:27.306275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.306072 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 11:11:27.306275 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.305984 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 11:11:27.311090 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.311071 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 11:11:27.312971 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.312947 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84cf84f4b5-gbjzw"] Jul 16 11:11:27.359815 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.359780 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/285179f9-23e7-470d-a659-49b1c5bc5a74-oauth-serving-cert\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.359815 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.359817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/285179f9-23e7-470d-a659-49b1c5bc5a74-trusted-ca-bundle\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.360023 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.359842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/285179f9-23e7-470d-a659-49b1c5bc5a74-console-config\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.360023 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.359893 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/285179f9-23e7-470d-a659-49b1c5bc5a74-console-oauth-config\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.360023 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.359947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx95v\" (UniqueName: \"kubernetes.io/projected/285179f9-23e7-470d-a659-49b1c5bc5a74-kube-api-access-kx95v\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.360023 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.359977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/285179f9-23e7-470d-a659-49b1c5bc5a74-console-serving-cert\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.360023 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.359993 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/285179f9-23e7-470d-a659-49b1c5bc5a74-service-ca\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.461155 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.461115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx95v\" (UniqueName: \"kubernetes.io/projected/285179f9-23e7-470d-a659-49b1c5bc5a74-kube-api-access-kx95v\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.461323 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.461217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/285179f9-23e7-470d-a659-49b1c5bc5a74-console-serving-cert\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.461323 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.461253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/285179f9-23e7-470d-a659-49b1c5bc5a74-service-ca\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.461323 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.461299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/285179f9-23e7-470d-a659-49b1c5bc5a74-oauth-serving-cert\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.461450 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.461343 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/285179f9-23e7-470d-a659-49b1c5bc5a74-trusted-ca-bundle\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.461450 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.461368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/285179f9-23e7-470d-a659-49b1c5bc5a74-console-config\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.461450 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.461403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/285179f9-23e7-470d-a659-49b1c5bc5a74-console-oauth-config\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.462158 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.462134 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/285179f9-23e7-470d-a659-49b1c5bc5a74-console-config\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.462260 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.462235 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/285179f9-23e7-470d-a659-49b1c5bc5a74-oauth-serving-cert\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.462308 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.462235 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/285179f9-23e7-470d-a659-49b1c5bc5a74-service-ca\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.462451 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.462429 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/285179f9-23e7-470d-a659-49b1c5bc5a74-trusted-ca-bundle\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.463824 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.463804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/285179f9-23e7-470d-a659-49b1c5bc5a74-console-oauth-config\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.463920 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.463890 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/285179f9-23e7-470d-a659-49b1c5bc5a74-console-serving-cert\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.476546 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.476518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx95v\" (UniqueName: \"kubernetes.io/projected/285179f9-23e7-470d-a659-49b1c5bc5a74-kube-api-access-kx95v\") pod \"console-84cf84f4b5-gbjzw\" (UID: \"285179f9-23e7-470d-a659-49b1c5bc5a74\") " pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.613025 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.612949 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:27.747489 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.747461 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84cf84f4b5-gbjzw"] Jul 16 11:11:27.750038 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:11:27.750010 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285179f9_23e7_470d_a659_49b1c5bc5a74.slice/crio-107503b375fbe9c423d0d49191125300d07cf121a256baf09dc9ebc41745f001 WatchSource:0}: Error finding container 107503b375fbe9c423d0d49191125300d07cf121a256baf09dc9ebc41745f001: Status 404 returned error can't find the container with id 107503b375fbe9c423d0d49191125300d07cf121a256baf09dc9ebc41745f001 Jul 16 11:11:27.891428 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.891347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cf84f4b5-gbjzw" event={"ID":"285179f9-23e7-470d-a659-49b1c5bc5a74","Type":"ContainerStarted","Data":"16c99de287e70dbeff739eed6412c41b3d330f8409ddfe34c4432e3e6df355c9"} Jul 16 11:11:27.891428 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.891394 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cf84f4b5-gbjzw" event={"ID":"285179f9-23e7-470d-a659-49b1c5bc5a74","Type":"ContainerStarted","Data":"107503b375fbe9c423d0d49191125300d07cf121a256baf09dc9ebc41745f001"} Jul 16 11:11:27.910126 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:27.910074 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84cf84f4b5-gbjzw" podStartSLOduration=0.910059649 podStartE2EDuration="910.059649ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:11:27.908502613 +0000 UTC m=+459.741851323" watchObservedRunningTime="2026-07-16 11:11:27.910059649 +0000 UTC m=+459.743408332" Jul 16 11:11:37.613618 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:37.613563 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:37.614080 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:37.613651 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:37.618362 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:37.618339 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:37.937266 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:37.937182 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84cf84f4b5-gbjzw" Jul 16 11:11:40.973735 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:40.973697 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:11:40.977142 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:40.977123 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-dzswm" Jul 16 11:11:40.979614 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:40.979591 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 16 11:11:40.979614 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:40.979603 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-l7crl\"" Jul 16 11:11:40.995184 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:40.995160 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:11:41.039615 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:41.039583 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:11:41.083927 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:41.083897 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq4vc\" (UniqueName: \"kubernetes.io/projected/41574ee0-18ac-4784-b08d-4d6f89cd7852-kube-api-access-qq4vc\") pod \"limitador-limitador-54d6d988d5-dzswm\" (UID: \"41574ee0-18ac-4784-b08d-4d6f89cd7852\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dzswm" Jul 16 11:11:41.084067 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:41.083950 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/41574ee0-18ac-4784-b08d-4d6f89cd7852-config-file\") pod \"limitador-limitador-54d6d988d5-dzswm\" (UID: \"41574ee0-18ac-4784-b08d-4d6f89cd7852\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dzswm" Jul 16 11:11:41.184883 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:41.184855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq4vc\" (UniqueName: \"kubernetes.io/projected/41574ee0-18ac-4784-b08d-4d6f89cd7852-kube-api-access-qq4vc\") pod \"limitador-limitador-54d6d988d5-dzswm\" (UID: \"41574ee0-18ac-4784-b08d-4d6f89cd7852\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dzswm" Jul 16 11:11:41.185054 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:41.184907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/41574ee0-18ac-4784-b08d-4d6f89cd7852-config-file\") pod \"limitador-limitador-54d6d988d5-dzswm\" (UID: \"41574ee0-18ac-4784-b08d-4d6f89cd7852\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dzswm" Jul 16 11:11:41.185481 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:41.185461 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/41574ee0-18ac-4784-b08d-4d6f89cd7852-config-file\") pod \"limitador-limitador-54d6d988d5-dzswm\" (UID: \"41574ee0-18ac-4784-b08d-4d6f89cd7852\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dzswm" Jul 16 11:11:41.194220 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:41.194199 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq4vc\" (UniqueName: \"kubernetes.io/projected/41574ee0-18ac-4784-b08d-4d6f89cd7852-kube-api-access-qq4vc\") pod \"limitador-limitador-54d6d988d5-dzswm\" (UID: \"41574ee0-18ac-4784-b08d-4d6f89cd7852\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dzswm" Jul 16 11:11:41.287178 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:41.287146 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-dzswm" Jul 16 11:11:41.424092 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:41.424069 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:11:41.426307 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:11:41.426282 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41574ee0_18ac_4784_b08d_4d6f89cd7852.slice/crio-b7347cdee36d6e1b1565c8690c8a030ed3dfa80670996a225b1da46f43d5631b WatchSource:0}: Error finding container b7347cdee36d6e1b1565c8690c8a030ed3dfa80670996a225b1da46f43d5631b: Status 404 returned error can't find the container with id b7347cdee36d6e1b1565c8690c8a030ed3dfa80670996a225b1da46f43d5631b Jul 16 11:11:41.950966 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:41.950933 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-dzswm" event={"ID":"41574ee0-18ac-4784-b08d-4d6f89cd7852","Type":"ContainerStarted","Data":"b7347cdee36d6e1b1565c8690c8a030ed3dfa80670996a225b1da46f43d5631b"} Jul 16 11:11:44.967824 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:44.967788 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-dzswm" event={"ID":"41574ee0-18ac-4784-b08d-4d6f89cd7852","Type":"ContainerStarted","Data":"12e609e17b7a511724194cf452d935f4e9b32886acafd136d754268bd7b6c6bf"} Jul 16 11:11:44.968194 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:44.967899 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-dzswm" Jul 16 11:11:44.988376 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:44.988328 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-dzswm" podStartSLOduration=1.65326346 podStartE2EDuration="4.988313539s" totalImagesPullingTime="3.335050079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:11:44.986535492 +0000 UTC m=+476.819884175" watchObservedRunningTime="2026-07-16 11:11:44.988313539 +0000 UTC m=+476.821662221" Jul 16 11:11:55.973533 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:11:55.973502 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-dzswm" Jul 16 11:12:59.318332 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:12:59.318254 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:13:15.010392 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:13:15.010355 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:13:22.191171 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:13:22.191130 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:13:34.096077 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:13:34.096040 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:13:42.731864 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:13:42.731829 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:13:48.629627 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:13:48.629600 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:13:48.630544 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:13:48.630522 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:13:57.517483 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:13:57.517449 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:14:03.112273 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.112237 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr"] Jul 16 11:14:03.115596 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.115566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.123892 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.123868 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 11:14:03.123892 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.123878 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 16 11:14:03.123892 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.123896 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 11:14:03.124113 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.123875 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xzgtr\"" Jul 16 11:14:03.136651 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.136627 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr"] Jul 16 11:14:03.219117 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.219080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.219281 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.219127 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.219281 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.219150 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bttjw\" (UniqueName: \"kubernetes.io/projected/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-kube-api-access-bttjw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.219281 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.219231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.219281 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.219262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.219281 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.219281 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.320705 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.320672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.320705 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.320707 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.320924 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.320723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.320924 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.320790 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.320924 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.320814 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.320924 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.320838 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bttjw\" (UniqueName: \"kubernetes.io/projected/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-kube-api-access-bttjw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.321120 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.321108 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.321217 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.321195 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.321280 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.321219 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.323068 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.323042 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.323242 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.323228 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.366296 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.366223 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bttjw\" (UniqueName: \"kubernetes.io/projected/da0e3b12-bceb-4cc1-aab0-d863cd4566f6-kube-api-access-bttjw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr\" (UID: \"da0e3b12-bceb-4cc1-aab0-d863cd4566f6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.425017 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.424981 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:03.591026 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.590997 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr"] Jul 16 11:14:03.592294 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:14:03.592273 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda0e3b12_bceb_4cc1_aab0_d863cd4566f6.slice/crio-511593c69e39cc65458df4fa66e804cc4e671df01749196a9fcf2e486a557fa9 WatchSource:0}: Error finding container 511593c69e39cc65458df4fa66e804cc4e671df01749196a9fcf2e486a557fa9: Status 404 returned error can't find the container with id 511593c69e39cc65458df4fa66e804cc4e671df01749196a9fcf2e486a557fa9 Jul 16 11:14:03.593621 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.593603 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 11:14:03.598672 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:03.598648 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" event={"ID":"da0e3b12-bceb-4cc1-aab0-d863cd4566f6","Type":"ContainerStarted","Data":"511593c69e39cc65458df4fa66e804cc4e671df01749196a9fcf2e486a557fa9"} Jul 16 11:14:05.609814 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:05.609778 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" event={"ID":"da0e3b12-bceb-4cc1-aab0-d863cd4566f6","Type":"ContainerStarted","Data":"b99a578c3e91e90faef7ad6be3de7c3ceb8511dd8021cbff1ba8be206e6be0f7"} Jul 16 11:14:05.635176 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:05.635124 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" podStartSLOduration=1.596481949 podStartE2EDuration="2.635104334s" totalImagesPullingTime="1.038622385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:14:05.633462368 +0000 UTC m=+617.466811051" watchObservedRunningTime="2026-07-16 11:14:05.635104334 +0000 UTC m=+617.468453028" Jul 16 11:14:06.127439 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:06.127401 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:14:13.425491 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:13.425414 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:13.425491 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:13.425453 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:13.438107 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:13.438082 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:14:13.659557 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:14:13.659530 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr" Jul 16 11:15:28.897563 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:15:28.897523 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:15:39.296897 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:15:39.296859 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:15:50.096200 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:15:50.096129 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:15:59.489195 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:15:59.489159 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:16:10.879507 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:16:10.879470 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:16:37.992866 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:16:37.992830 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:16:51.202081 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:16:51.202044 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:17:29.988063 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:17:29.988026 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:17:39.085867 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:17:39.085830 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:17:47.891197 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:17:47.891160 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:17:59.289430 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:17:59.289396 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:18:09.614554 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:18:09.614518 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:18:21.292115 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:18:21.292083 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:18:32.114852 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:18:32.114813 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:18:42.192618 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:18:42.192521 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:18:48.659086 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:18:48.659052 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:18:48.659523 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:18:48.659273 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:19:07.194730 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:19:07.194699 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:19:16.887157 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:19:16.887122 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:19:39.585818 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:19:39.585786 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:19:50.702300 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:19:50.702266 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:20:08.256772 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:20:08.256729 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:20:17.786970 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:20:17.786930 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:20:27.501460 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:20:27.501427 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:20:37.690624 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:20:37.690568 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:20:47.495148 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:20:47.495111 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:22:35.090694 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:22:35.090660 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:22:46.194149 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:22:46.194109 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:22:55.084035 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:22:55.084004 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:23:07.698417 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:23:07.698379 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:23:17.384510 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:23:17.384476 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:23:28.887313 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:23:28.887283 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:23:37.886897 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:23:37.886862 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:23:48.687795 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:23:48.687771 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:23:48.688265 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:23:48.688131 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:23:49.794703 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:23:49.794671 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:23:59.595161 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:23:59.595119 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:24:08.579672 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:24:08.579638 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:24:22.180748 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:24:22.180688 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:24:39.495734 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:24:39.495694 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:24:48.703644 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:24:48.703560 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:24:59.197466 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:24:59.197431 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:25:08.690008 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:25:08.689977 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:25:29.292873 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:25:29.292840 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:25:41.592681 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:25:41.592652 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:25:51.193793 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:25:51.193754 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:26:04.597567 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:26:04.597530 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:26:14.088166 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:26:14.088082 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:26:27.798060 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:26:27.798024 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:26:37.689934 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:26:37.689887 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:26:52.703486 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:26:52.703452 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:27:01.601541 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:27:01.601506 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:27:12.889849 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:27:12.889811 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:27:32.102452 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:27:32.102418 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:27:43.294227 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:27:43.294131 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:27:55.890835 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:27:55.890801 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:28:06.019884 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:28:06.019847 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:28:19.387615 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:28:19.387567 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:28:32.188043 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:28:32.187970 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:28:40.586600 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:28:40.586554 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:28:48.716673 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:28:48.716645 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:28:48.718213 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:28:48.718193 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:28:50.784499 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:28:50.784461 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:28:59.687101 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:28:59.687070 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:29:09.481945 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:29:09.481909 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:29:18.387478 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:29:18.387448 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:30:47.513269 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:30:47.513181 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:30:54.791831 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:30:54.791798 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:31:03.887620 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:31:03.887567 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:31:14.179919 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:31:14.179882 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:31:25.890643 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:31:25.890591 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:31:56.683809 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:31:56.683773 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:32:09.003048 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:32:09.003010 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:32:57.901830 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:32:57.901796 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:33:08.679662 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:08.679627 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:33:32.186825 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.186793 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq"] Jul 16 11:33:32.189961 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.189942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.193929 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.193905 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bb5ebc\"/\"default-dockercfg-sdbww\"" Jul 16 11:33:32.193929 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.193921 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bb5ebc\"/\"kube-root-ca.crt\"" Jul 16 11:33:32.194097 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.193971 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bb5ebc\"/\"openshift-service-ca.crt\"" Jul 16 11:33:32.194097 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.194014 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bb5ebc\"/\"test-model-bb5ebc-kserve-self-signed-certs\"" Jul 16 11:33:32.200294 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.200274 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq"] Jul 16 11:33:32.367802 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.367773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-model-cache\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.367802 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.367805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r69h\" (UniqueName: \"kubernetes.io/projected/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-kube-api-access-6r69h\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.368032 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.367844 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-home\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.368032 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.367940 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-tmp-dir\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.368032 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.367963 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-tls-certs\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.368032 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.367984 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-dshm\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.469072 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.468985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-model-cache\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.469072 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.469025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r69h\" (UniqueName: \"kubernetes.io/projected/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-kube-api-access-6r69h\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.469285 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.469088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-home\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.469285 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.469121 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-tmp-dir\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.469285 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.469141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-tls-certs\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.469285 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.469163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-dshm\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.469431 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.469417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-model-cache\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.469465 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.469446 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-home\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.469503 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.469479 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-tmp-dir\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.471394 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.471370 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-dshm\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.471557 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.471542 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-tls-certs\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.479278 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.479254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r69h\" (UniqueName: \"kubernetes.io/projected/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-kube-api-access-6r69h\") pod \"test-model-bb5ebc-kserve-594dfbc595-lrplq\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.499286 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.499264 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:32.630779 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.630757 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq"] Jul 16 11:33:32.632387 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:33:32.632360 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe52cf2_a5d5_4d40_95c5_9f51b8bc1125.slice/crio-f7cb2bbd235b8828917f00f994a8a061976c5c1abb557607e609d2945eeb6beb WatchSource:0}: Error finding container f7cb2bbd235b8828917f00f994a8a061976c5c1abb557607e609d2945eeb6beb: Status 404 returned error can't find the container with id f7cb2bbd235b8828917f00f994a8a061976c5c1abb557607e609d2945eeb6beb Jul 16 11:33:32.932714 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.932681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" event={"ID":"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125","Type":"ContainerStarted","Data":"73ec0c6195d3e11aeb7dd985ed96066fe59dc255ae424002e2e6ca70c04d1658"} Jul 16 11:33:32.932714 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.932720 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" event={"ID":"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125","Type":"ContainerStarted","Data":"f7cb2bbd235b8828917f00f994a8a061976c5c1abb557607e609d2945eeb6beb"} Jul 16 11:33:32.953324 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:32.953280 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" podStartSLOduration=0.953262504 podStartE2EDuration="953.262504ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:33:32.951482669 +0000 UTC m=+1784.784831350" watchObservedRunningTime="2026-07-16 11:33:32.953262504 +0000 UTC m=+1784.786611200" Jul 16 11:33:35.685824 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:35.685790 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:33:42.499425 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:42.499333 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:42.499425 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:42.499405 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:42.511850 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:42.511827 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:42.992120 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:42.992044 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:33:47.682472 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:47.682432 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:33:48.744364 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:48.744333 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:33:48.746878 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:33:48.746858 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:34:01.978732 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:01.978695 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx"] Jul 16 11:34:01.981968 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:01.981942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:01.985821 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:01.985802 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-96e676\"/\"default-dockercfg-s7w9w\"" Jul 16 11:34:01.985931 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:01.985882 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-96e676\"/\"openshift-service-ca.crt\"" Jul 16 11:34:01.985931 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:01.985883 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-96e676\"/\"test-model-96e676-kserve-self-signed-certs\"" Jul 16 11:34:01.986008 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:01.985935 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-96e676\"/\"kube-root-ca.crt\"" Jul 16 11:34:02.005500 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.005472 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx"] Jul 16 11:34:02.023674 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.023643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-dshm\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.023674 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.023673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-tmp-dir\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.023840 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.023692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnbnp\" (UniqueName: \"kubernetes.io/projected/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-kube-api-access-jnbnp\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.023840 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.023814 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-model-cache\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.023959 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.023847 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-home\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.023959 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.023876 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-tls-certs\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.124615 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.124585 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-model-cache\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.124789 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.124623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-home\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.124789 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.124736 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-tls-certs\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.124879 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.124811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-dshm\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.124879 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.124829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-tmp-dir\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.124879 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.124845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnbnp\" (UniqueName: \"kubernetes.io/projected/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-kube-api-access-jnbnp\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.125011 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.124995 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-home\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.125060 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.125019 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-model-cache\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.125203 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.125182 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-tmp-dir\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.126933 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.126912 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-dshm\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.127229 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.127212 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-tls-certs\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.133709 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.133687 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnbnp\" (UniqueName: \"kubernetes.io/projected/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-kube-api-access-jnbnp\") pod \"test-model-96e676-kserve-759967f595-sp7nx\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.291128 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.291093 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:02.424234 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:02.424115 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx"] Jul 16 11:34:02.427241 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:34:02.427211 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72f6d2cd_ffad_4fd0_bd6b_087b6e73a3e1.slice/crio-a5201e0e7bb8e80af275f0d27ccb7b1615b6c3a3b7fb0dd7c547c69cac37b2a6 WatchSource:0}: Error finding container a5201e0e7bb8e80af275f0d27ccb7b1615b6c3a3b7fb0dd7c547c69cac37b2a6: Status 404 returned error can't find the container with id a5201e0e7bb8e80af275f0d27ccb7b1615b6c3a3b7fb0dd7c547c69cac37b2a6 Jul 16 11:34:03.068945 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:03.068912 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" event={"ID":"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1","Type":"ContainerStarted","Data":"2a926c5f432da3a0f029d7e1c779071c0bff7973c94189a5bf5f67a6836851e8"} Jul 16 11:34:03.068945 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:03.068951 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" event={"ID":"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1","Type":"ContainerStarted","Data":"a5201e0e7bb8e80af275f0d27ccb7b1615b6c3a3b7fb0dd7c547c69cac37b2a6"} Jul 16 11:34:03.090353 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:03.090304 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" podStartSLOduration=2.090286827 podStartE2EDuration="2.090286827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:34:03.089422596 +0000 UTC m=+1814.922771278" watchObservedRunningTime="2026-07-16 11:34:03.090286827 +0000 UTC m=+1814.923635532" Jul 16 11:34:06.193555 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:06.193521 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:34:12.291979 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:12.291943 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:12.294430 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:12.292003 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:12.304458 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:12.304433 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:13.126314 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:13.126287 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:20.585582 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:20.585536 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:34:27.286896 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:27.286859 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq"] Jul 16 11:34:28.333103 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:28.333066 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx"] Jul 16 11:34:34.197666 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:34.197631 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:34:42.307162 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.307104 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" podUID="1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125" containerName="main" containerID="cri-o://73ec0c6195d3e11aeb7dd985ed96066fe59dc255ae424002e2e6ca70c04d1658" gracePeriod=45 Jul 16 11:34:42.555140 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.555118 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:34:42.579604 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.579517 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-model-cache\") pod \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " Jul 16 11:34:42.579604 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.579596 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-tls-certs\") pod \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " Jul 16 11:34:42.579805 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.579622 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-dshm\") pod \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " Jul 16 11:34:42.579805 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.579659 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-tmp-dir\") pod \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " Jul 16 11:34:42.579805 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.579704 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r69h\" (UniqueName: \"kubernetes.io/projected/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-kube-api-access-6r69h\") pod \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " Jul 16 11:34:42.579805 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.579773 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-home\") pod \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\" (UID: \"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125\") " Jul 16 11:34:42.580008 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.579826 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-model-cache" (OuterVolumeSpecName: "model-cache") pod "1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125" (UID: "1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:34:42.580072 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.580016 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125" (UID: "1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:34:42.580125 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.580081 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-model-cache\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:34:42.580271 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.580237 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-home" (OuterVolumeSpecName: "home") pod "1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125" (UID: "1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:34:42.581803 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.581773 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-dshm" (OuterVolumeSpecName: "dshm") pod "1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125" (UID: "1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:34:42.582009 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.581986 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-kube-api-access-6r69h" (OuterVolumeSpecName: "kube-api-access-6r69h") pod "1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125" (UID: "1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125"). InnerVolumeSpecName "kube-api-access-6r69h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:34:42.582170 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.582147 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125" (UID: "1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:34:42.681359 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.681328 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-home\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:34:42.681359 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.681355 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-tls-certs\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:34:42.681359 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.681365 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-dshm\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:34:42.681673 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.681373 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-tmp-dir\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:34:42.681673 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:42.681381 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r69h\" (UniqueName: \"kubernetes.io/projected/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125-kube-api-access-6r69h\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:34:43.255255 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.255226 2580 generic.go:356] "Generic (PLEG): container finished" podID="1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125" containerID="73ec0c6195d3e11aeb7dd985ed96066fe59dc255ae424002e2e6ca70c04d1658" exitCode=0 Jul 16 11:34:43.255440 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.255269 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" Jul 16 11:34:43.255440 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.255302 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" event={"ID":"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125","Type":"ContainerDied","Data":"73ec0c6195d3e11aeb7dd985ed96066fe59dc255ae424002e2e6ca70c04d1658"} Jul 16 11:34:43.255440 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.255335 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq" event={"ID":"1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125","Type":"ContainerDied","Data":"f7cb2bbd235b8828917f00f994a8a061976c5c1abb557607e609d2945eeb6beb"} Jul 16 11:34:43.255440 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.255351 2580 scope.go:117] "RemoveContainer" containerID="73ec0c6195d3e11aeb7dd985ed96066fe59dc255ae424002e2e6ca70c04d1658" Jul 16 11:34:43.264092 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.264071 2580 scope.go:117] "RemoveContainer" containerID="73ec0c6195d3e11aeb7dd985ed96066fe59dc255ae424002e2e6ca70c04d1658" Jul 16 11:34:43.264356 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:34:43.264339 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ec0c6195d3e11aeb7dd985ed96066fe59dc255ae424002e2e6ca70c04d1658\": container with ID starting with 73ec0c6195d3e11aeb7dd985ed96066fe59dc255ae424002e2e6ca70c04d1658 not found: ID does not exist" containerID="73ec0c6195d3e11aeb7dd985ed96066fe59dc255ae424002e2e6ca70c04d1658" Jul 16 11:34:43.264409 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.264363 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ec0c6195d3e11aeb7dd985ed96066fe59dc255ae424002e2e6ca70c04d1658"} err="failed to get container status \"73ec0c6195d3e11aeb7dd985ed96066fe59dc255ae424002e2e6ca70c04d1658\": rpc error: code = NotFound desc = could not find container \"73ec0c6195d3e11aeb7dd985ed96066fe59dc255ae424002e2e6ca70c04d1658\": container with ID starting with 73ec0c6195d3e11aeb7dd985ed96066fe59dc255ae424002e2e6ca70c04d1658 not found: ID does not exist" Jul 16 11:34:43.275798 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.275771 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq"] Jul 16 11:34:43.280884 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.280865 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bb5ebc/test-model-bb5ebc-kserve-594dfbc595-lrplq"] Jul 16 11:34:43.352916 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.352879 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" podUID="72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1" containerName="main" containerID="cri-o://2a926c5f432da3a0f029d7e1c779071c0bff7973c94189a5bf5f67a6836851e8" gracePeriod=45 Jul 16 11:34:43.591349 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.591326 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:43.689310 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.689277 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-model-cache\") pod \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " Jul 16 11:34:43.689310 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.689315 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-tls-certs\") pod \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " Jul 16 11:34:43.689507 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.689356 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-tmp-dir\") pod \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " Jul 16 11:34:43.689507 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.689398 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-dshm\") pod \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " Jul 16 11:34:43.689616 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.689559 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1" (UID: "72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:34:43.689664 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.689632 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-home\") pod \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " Jul 16 11:34:43.689706 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.689659 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1" (UID: "72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:34:43.689706 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.689666 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnbnp\" (UniqueName: \"kubernetes.io/projected/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-kube-api-access-jnbnp\") pod \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\" (UID: \"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1\") " Jul 16 11:34:43.689870 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.689844 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-home" (OuterVolumeSpecName: "home") pod "72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1" (UID: "72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:34:43.690022 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.690007 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-tmp-dir\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:34:43.690088 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.690025 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-home\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:34:43.690088 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.690034 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-model-cache\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:34:43.691671 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.691646 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1" (UID: "72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:34:43.691776 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.691686 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-dshm" (OuterVolumeSpecName: "dshm") pod "72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1" (UID: "72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:34:43.691776 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.691717 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-kube-api-access-jnbnp" (OuterVolumeSpecName: "kube-api-access-jnbnp") pod "72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1" (UID: "72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1"). InnerVolumeSpecName "kube-api-access-jnbnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:34:43.791025 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.790931 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-dshm\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:34:43.791025 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.790961 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnbnp\" (UniqueName: \"kubernetes.io/projected/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-kube-api-access-jnbnp\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:34:43.791025 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:43.790975 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1-tls-certs\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:34:44.261758 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:44.261725 2580 generic.go:356] "Generic (PLEG): container finished" podID="72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1" containerID="2a926c5f432da3a0f029d7e1c779071c0bff7973c94189a5bf5f67a6836851e8" exitCode=0 Jul 16 11:34:44.261952 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:44.261770 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" Jul 16 11:34:44.261952 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:44.261795 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" event={"ID":"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1","Type":"ContainerDied","Data":"2a926c5f432da3a0f029d7e1c779071c0bff7973c94189a5bf5f67a6836851e8"} Jul 16 11:34:44.261952 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:44.261833 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx" event={"ID":"72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1","Type":"ContainerDied","Data":"a5201e0e7bb8e80af275f0d27ccb7b1615b6c3a3b7fb0dd7c547c69cac37b2a6"} Jul 16 11:34:44.261952 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:44.261850 2580 scope.go:117] "RemoveContainer" containerID="2a926c5f432da3a0f029d7e1c779071c0bff7973c94189a5bf5f67a6836851e8" Jul 16 11:34:44.270232 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:44.270213 2580 scope.go:117] "RemoveContainer" containerID="2a926c5f432da3a0f029d7e1c779071c0bff7973c94189a5bf5f67a6836851e8" Jul 16 11:34:44.270483 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:34:44.270461 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a926c5f432da3a0f029d7e1c779071c0bff7973c94189a5bf5f67a6836851e8\": container with ID starting with 2a926c5f432da3a0f029d7e1c779071c0bff7973c94189a5bf5f67a6836851e8 not found: ID does not exist" containerID="2a926c5f432da3a0f029d7e1c779071c0bff7973c94189a5bf5f67a6836851e8" Jul 16 11:34:44.270549 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:44.270492 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a926c5f432da3a0f029d7e1c779071c0bff7973c94189a5bf5f67a6836851e8"} err="failed to get container status \"2a926c5f432da3a0f029d7e1c779071c0bff7973c94189a5bf5f67a6836851e8\": rpc error: code = NotFound desc = could not find container \"2a926c5f432da3a0f029d7e1c779071c0bff7973c94189a5bf5f67a6836851e8\": container with ID starting with 2a926c5f432da3a0f029d7e1c779071c0bff7973c94189a5bf5f67a6836851e8 not found: ID does not exist" Jul 16 11:34:44.289629 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:44.289595 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx"] Jul 16 11:34:44.295333 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:44.295308 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-96e676/test-model-96e676-kserve-759967f595-sp7nx"] Jul 16 11:34:44.299633 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:34:44.299612 2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72f6d2cd_ffad_4fd0_bd6b_087b6e73a3e1.slice\": RecentStats: unable to find data in memory cache]" Jul 16 11:34:44.886528 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:44.886495 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125" path="/var/lib/kubelet/pods/1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125/volumes" Jul 16 11:34:44.886920 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:34:44.886905 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1" path="/var/lib/kubelet/pods/72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1/volumes" Jul 16 11:35:13.692124 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:35:13.692050 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:35:27.485800 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:35:27.485763 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:35:37.786357 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:35:37.786315 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:35:52.796119 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:35:52.796081 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:36:07.587243 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:36:07.587209 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:36:20.088225 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:36:20.088182 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:36:31.692099 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:36:31.692058 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:36:46.021006 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:36:46.020924 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:37:00.290980 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.290943 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b"] Jul 16 11:37:00.291329 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.291310 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125" containerName="main" Jul 16 11:37:00.291329 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.291321 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125" containerName="main" Jul 16 11:37:00.291401 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.291345 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1" containerName="main" Jul 16 11:37:00.291401 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.291353 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1" containerName="main" Jul 16 11:37:00.291465 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.291406 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72f6d2cd-ffad-4fd0-bd6b-087b6e73a3e1" containerName="main" Jul 16 11:37:00.291465 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.291416 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe52cf2-a5d5-4d40-95c5-9f51b8bc1125" containerName="main" Jul 16 11:37:00.294112 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.294095 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.298097 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.298069 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8bae7d\"/\"auth-test-model-8bae7d-kserve-self-signed-certs\"" Jul 16 11:37:00.298222 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.298070 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8bae7d\"/\"default-dockercfg-hxjpd\"" Jul 16 11:37:00.298222 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.298144 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8bae7d\"/\"openshift-service-ca.crt\"" Jul 16 11:37:00.298222 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.298171 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8bae7d\"/\"kube-root-ca.crt\"" Jul 16 11:37:00.303843 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.303816 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b"] Jul 16 11:37:00.322935 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.322904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-tmp-dir\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.323057 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.322943 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-model-cache\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.323057 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.322996 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc06635c-1e56-446d-9804-c64c54a83e42-tls-certs\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.323057 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.323016 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8zwp\" (UniqueName: \"kubernetes.io/projected/bc06635c-1e56-446d-9804-c64c54a83e42-kube-api-access-r8zwp\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.323161 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.323086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-dshm\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.323161 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.323106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-home\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.423472 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.423446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-home\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.423729 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.423490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-tmp-dir\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.423729 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.423515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-model-cache\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.423729 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.423644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc06635c-1e56-446d-9804-c64c54a83e42-tls-certs\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.423729 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.423693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8zwp\" (UniqueName: \"kubernetes.io/projected/bc06635c-1e56-446d-9804-c64c54a83e42-kube-api-access-r8zwp\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.423960 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.423793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-dshm\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.423960 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.423886 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-tmp-dir\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.423960 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.423936 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-model-cache\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.424084 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.423988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-home\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.425891 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.425866 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-dshm\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.426099 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.426080 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc06635c-1e56-446d-9804-c64c54a83e42-tls-certs\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.432477 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.432447 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8zwp\" (UniqueName: \"kubernetes.io/projected/bc06635c-1e56-446d-9804-c64c54a83e42-kube-api-access-r8zwp\") pod \"auth-test-model-8bae7d-kserve-6749469dd7-84h8b\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.605246 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.605169 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:00.737243 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.737219 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b"] Jul 16 11:37:00.739284 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:37:00.739256 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc06635c_1e56_446d_9804_c64c54a83e42.slice/crio-95e1eb2da68cb5bdf7074e2b017eb12fc53290ceefe47f66bf9a256ed197de8f WatchSource:0}: Error finding container 95e1eb2da68cb5bdf7074e2b017eb12fc53290ceefe47f66bf9a256ed197de8f: Status 404 returned error can't find the container with id 95e1eb2da68cb5bdf7074e2b017eb12fc53290ceefe47f66bf9a256ed197de8f Jul 16 11:37:00.893766 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.893681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" event={"ID":"bc06635c-1e56-446d-9804-c64c54a83e42","Type":"ContainerStarted","Data":"967a0d39df0730411c9562804ff4dce1957f9e202c9492b82cce85f89060c308"} Jul 16 11:37:00.893766 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.893721 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" event={"ID":"bc06635c-1e56-446d-9804-c64c54a83e42","Type":"ContainerStarted","Data":"95e1eb2da68cb5bdf7074e2b017eb12fc53290ceefe47f66bf9a256ed197de8f"} Jul 16 11:37:00.910669 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:00.910617 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" podStartSLOduration=0.910600481 podStartE2EDuration="910.600481ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:37:00.909343942 +0000 UTC m=+1992.742692639" watchObservedRunningTime="2026-07-16 11:37:00.910600481 +0000 UTC m=+1992.743949157" Jul 16 11:37:02.403244 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:02.403208 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:37:10.605778 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:10.605745 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:10.606248 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:10.605821 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:10.618322 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:10.618298 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:10.949342 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:10.949263 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:37:25.565831 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.565799 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt"] Jul 16 11:37:25.569110 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.569090 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.572900 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.572876 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a47fa3\"/\"openshift-service-ca.crt\"" Jul 16 11:37:25.573037 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.572876 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a47fa3\"/\"auth-test-model-a47fa3-kserve-self-signed-certs\"" Jul 16 11:37:25.573037 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.572925 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a47fa3\"/\"default-dockercfg-4vw9q\"" Jul 16 11:37:25.573037 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.572928 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a47fa3\"/\"kube-root-ca.crt\"" Jul 16 11:37:25.579858 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.579831 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt"] Jul 16 11:37:25.623023 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.622992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-dshm\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.623023 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.623023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-tmp-dir\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.623214 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.623057 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-model-cache\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.623214 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.623101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr4rs\" (UniqueName: \"kubernetes.io/projected/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-kube-api-access-cr4rs\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.623214 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.623122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-home\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.623214 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.623197 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-tls-certs\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.724003 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.723957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4rs\" (UniqueName: \"kubernetes.io/projected/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-kube-api-access-cr4rs\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.724169 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.724012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-home\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.724218 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.724180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-tls-certs\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.724267 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.724244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-dshm\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.724267 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.724262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-tmp-dir\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.724367 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.724313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-model-cache\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.724431 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.724409 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-home\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.724634 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.724605 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-tmp-dir\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.724749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.724651 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-model-cache\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.726415 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.726397 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-dshm\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.726671 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.726656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-tls-certs\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.732824 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.732803 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4rs\" (UniqueName: \"kubernetes.io/projected/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-kube-api-access-cr4rs\") pod \"auth-test-model-a47fa3-kserve-854c8c6487-bbptt\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:25.879349 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:25.879261 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:26.010797 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:26.010767 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt"] Jul 16 11:37:26.013135 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:37:26.013109 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2652e4d4_d2c3_41f1_ab38_61e9895fa0a6.slice/crio-19e1753d25efde97e478f396fbda0b2ab5994487e64347dfe052e094840ddbd1 WatchSource:0}: Error finding container 19e1753d25efde97e478f396fbda0b2ab5994487e64347dfe052e094840ddbd1: Status 404 returned error can't find the container with id 19e1753d25efde97e478f396fbda0b2ab5994487e64347dfe052e094840ddbd1 Jul 16 11:37:27.009355 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:27.009322 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" event={"ID":"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6","Type":"ContainerStarted","Data":"88cc8e9697f864d266625289c10ada3225a5ee2f74aba8d458daa7eaf0353dc0"} Jul 16 11:37:27.009355 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:27.009359 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" event={"ID":"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6","Type":"ContainerStarted","Data":"19e1753d25efde97e478f396fbda0b2ab5994487e64347dfe052e094840ddbd1"} Jul 16 11:37:27.029138 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:27.029096 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" podStartSLOduration=2.029080583 podStartE2EDuration="2.029080583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:37:27.027106411 +0000 UTC m=+2018.860455094" watchObservedRunningTime="2026-07-16 11:37:27.029080583 +0000 UTC m=+2018.862429264" Jul 16 11:37:29.086008 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:29.085963 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:37:35.879686 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:35.879642 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:35.880151 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:35.879719 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:35.892197 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:35.892176 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:36.064482 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:36.064457 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:37:51.383014 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:37:51.382981 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:38:10.081474 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:38:10.081437 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:38:26.582964 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:38:26.582920 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:38:43.613659 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:38:43.613624 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:38:48.771803 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:38:48.771777 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:38:48.776301 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:38:48.776278 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66mkq_04bb84b9-bd39-419b-947a-2ab4c9d0476d/ovn-acl-logging/0.log" Jul 16 11:39:01.088591 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:01.088551 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:39:15.293957 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:15.293922 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:39:33.285553 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:33.285519 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:39:38.731337 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:38.731297 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b"] Jul 16 11:39:39.696603 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:39.696557 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt"] Jul 16 11:39:44.285094 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.285017 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd"] Jul 16 11:39:44.288460 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.288443 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.291263 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.291241 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8bae7d\"/\"sub-test-model-8bae7d-kserve-self-signed-certs\"" Jul 16 11:39:44.300912 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.300859 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd"] Jul 16 11:39:44.379705 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.379667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23d07b4-95be-4eb3-92ae-b2a03769ec93-tls-certs\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.379882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.379745 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rhv6\" (UniqueName: \"kubernetes.io/projected/e23d07b4-95be-4eb3-92ae-b2a03769ec93-kube-api-access-8rhv6\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.379882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.379771 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-home\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.379882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.379812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-tmp-dir\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.379882 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.379862 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-dshm\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.380014 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.379901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-model-cache\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.481089 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.481050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-tmp-dir\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.481089 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.481084 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-dshm\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.481089 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.481100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-model-cache\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.481361 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.481121 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23d07b4-95be-4eb3-92ae-b2a03769ec93-tls-certs\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.481361 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.481256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rhv6\" (UniqueName: \"kubernetes.io/projected/e23d07b4-95be-4eb3-92ae-b2a03769ec93-kube-api-access-8rhv6\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.481361 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.481308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-home\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.481637 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.481528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-tmp-dir\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.481637 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.481614 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-model-cache\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.481753 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.481681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-home\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.483403 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.483379 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-dshm\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.483660 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.483644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23d07b4-95be-4eb3-92ae-b2a03769ec93-tls-certs\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.490393 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.490372 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rhv6\" (UniqueName: \"kubernetes.io/projected/e23d07b4-95be-4eb3-92ae-b2a03769ec93-kube-api-access-8rhv6\") pod \"sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.597965 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.597885 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:44.732552 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:44.732511 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd"] Jul 16 11:39:44.739864 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:39:44.739833 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode23d07b4_95be_4eb3_92ae_b2a03769ec93.slice/crio-46ee8503790f2c5c9714a6f0b9836b8c4e8545e346557b87e152300a3a79073d WatchSource:0}: Error finding container 46ee8503790f2c5c9714a6f0b9836b8c4e8545e346557b87e152300a3a79073d: Status 404 returned error can't find the container with id 46ee8503790f2c5c9714a6f0b9836b8c4e8545e346557b87e152300a3a79073d Jul 16 11:39:45.662838 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:45.662803 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" event={"ID":"e23d07b4-95be-4eb3-92ae-b2a03769ec93","Type":"ContainerStarted","Data":"033ffe6435062acc49225ef480dfaa0b96cf8184c7908030e8de6ade1cca2c10"} Jul 16 11:39:45.662838 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:45.662845 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" event={"ID":"e23d07b4-95be-4eb3-92ae-b2a03769ec93","Type":"ContainerStarted","Data":"46ee8503790f2c5c9714a6f0b9836b8c4e8545e346557b87e152300a3a79073d"} Jul 16 11:39:45.692167 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:45.692118 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" podStartSLOduration=1.6921029920000001 podStartE2EDuration="1.692102992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:39:45.6906987 +0000 UTC m=+2157.524047382" watchObservedRunningTime="2026-07-16 11:39:45.692102992 +0000 UTC m=+2157.525451668" Jul 16 11:39:52.483926 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:52.483886 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:39:53.751023 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:53.750959 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" podUID="bc06635c-1e56-446d-9804-c64c54a83e42" containerName="main" containerID="cri-o://967a0d39df0730411c9562804ff4dce1957f9e202c9492b82cce85f89060c308" gracePeriod=45 Jul 16 11:39:54.001520 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.001463 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:39:54.071661 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.071633 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8zwp\" (UniqueName: \"kubernetes.io/projected/bc06635c-1e56-446d-9804-c64c54a83e42-kube-api-access-r8zwp\") pod \"bc06635c-1e56-446d-9804-c64c54a83e42\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " Jul 16 11:39:54.071853 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.071704 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-model-cache\") pod \"bc06635c-1e56-446d-9804-c64c54a83e42\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " Jul 16 11:39:54.071853 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.071728 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc06635c-1e56-446d-9804-c64c54a83e42-tls-certs\") pod \"bc06635c-1e56-446d-9804-c64c54a83e42\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " Jul 16 11:39:54.071853 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.071825 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-dshm\") pod \"bc06635c-1e56-446d-9804-c64c54a83e42\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " Jul 16 11:39:54.072023 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.071913 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc06635c-1e56-446d-9804-c64c54a83e42" (UID: "bc06635c-1e56-446d-9804-c64c54a83e42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:39:54.072023 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.071923 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-tmp-dir\") pod \"bc06635c-1e56-446d-9804-c64c54a83e42\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " Jul 16 11:39:54.072023 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.072017 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-home\") pod \"bc06635c-1e56-446d-9804-c64c54a83e42\" (UID: \"bc06635c-1e56-446d-9804-c64c54a83e42\") " Jul 16 11:39:54.072157 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.072097 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc06635c-1e56-446d-9804-c64c54a83e42" (UID: "bc06635c-1e56-446d-9804-c64c54a83e42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:39:54.072300 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.072277 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-home" (OuterVolumeSpecName: "home") pod "bc06635c-1e56-446d-9804-c64c54a83e42" (UID: "bc06635c-1e56-446d-9804-c64c54a83e42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:39:54.072389 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.072372 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-home\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:39:54.072448 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.072395 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-model-cache\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:39:54.072448 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.072407 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-tmp-dir\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:39:54.073918 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.073891 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-dshm" (OuterVolumeSpecName: "dshm") pod "bc06635c-1e56-446d-9804-c64c54a83e42" (UID: "bc06635c-1e56-446d-9804-c64c54a83e42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:39:54.073918 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.073900 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc06635c-1e56-446d-9804-c64c54a83e42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc06635c-1e56-446d-9804-c64c54a83e42" (UID: "bc06635c-1e56-446d-9804-c64c54a83e42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:39:54.074036 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.073985 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc06635c-1e56-446d-9804-c64c54a83e42-kube-api-access-r8zwp" (OuterVolumeSpecName: "kube-api-access-r8zwp") pod "bc06635c-1e56-446d-9804-c64c54a83e42" (UID: "bc06635c-1e56-446d-9804-c64c54a83e42"). InnerVolumeSpecName "kube-api-access-r8zwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:39:54.173761 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.173730 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8zwp\" (UniqueName: \"kubernetes.io/projected/bc06635c-1e56-446d-9804-c64c54a83e42-kube-api-access-r8zwp\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:39:54.173761 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.173756 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc06635c-1e56-446d-9804-c64c54a83e42-tls-certs\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:39:54.173761 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.173767 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc06635c-1e56-446d-9804-c64c54a83e42-dshm\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:39:54.598850 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.598815 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:54.599123 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.599097 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:54.611350 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.611326 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:54.706782 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.706741 2580 generic.go:356] "Generic (PLEG): container finished" podID="bc06635c-1e56-446d-9804-c64c54a83e42" containerID="967a0d39df0730411c9562804ff4dce1957f9e202c9492b82cce85f89060c308" exitCode=0 Jul 16 11:39:54.706949 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.706788 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" Jul 16 11:39:54.706949 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.706826 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" event={"ID":"bc06635c-1e56-446d-9804-c64c54a83e42","Type":"ContainerDied","Data":"967a0d39df0730411c9562804ff4dce1957f9e202c9492b82cce85f89060c308"} Jul 16 11:39:54.706949 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.706873 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b" event={"ID":"bc06635c-1e56-446d-9804-c64c54a83e42","Type":"ContainerDied","Data":"95e1eb2da68cb5bdf7074e2b017eb12fc53290ceefe47f66bf9a256ed197de8f"} Jul 16 11:39:54.706949 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.706894 2580 scope.go:117] "RemoveContainer" containerID="967a0d39df0730411c9562804ff4dce1957f9e202c9492b82cce85f89060c308" Jul 16 11:39:54.715605 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.715568 2580 scope.go:117] "RemoveContainer" containerID="967a0d39df0730411c9562804ff4dce1957f9e202c9492b82cce85f89060c308" Jul 16 11:39:54.715966 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:39:54.715943 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967a0d39df0730411c9562804ff4dce1957f9e202c9492b82cce85f89060c308\": container with ID starting with 967a0d39df0730411c9562804ff4dce1957f9e202c9492b82cce85f89060c308 not found: ID does not exist" containerID="967a0d39df0730411c9562804ff4dce1957f9e202c9492b82cce85f89060c308" Jul 16 11:39:54.716055 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.715976 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967a0d39df0730411c9562804ff4dce1957f9e202c9492b82cce85f89060c308"} err="failed to get container status \"967a0d39df0730411c9562804ff4dce1957f9e202c9492b82cce85f89060c308\": rpc error: code = NotFound desc = could not find container \"967a0d39df0730411c9562804ff4dce1957f9e202c9492b82cce85f89060c308\": container with ID starting with 967a0d39df0730411c9562804ff4dce1957f9e202c9492b82cce85f89060c308 not found: ID does not exist" Jul 16 11:39:54.716877 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.716822 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" podUID="2652e4d4-d2c3-41f1-ab38-61e9895fa0a6" containerName="main" containerID="cri-o://88cc8e9697f864d266625289c10ada3225a5ee2f74aba8d458daa7eaf0353dc0" gracePeriod=45 Jul 16 11:39:54.722029 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.722010 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:39:54.732637 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.732588 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b"] Jul 16 11:39:54.734549 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.734528 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/auth-test-model-8bae7d-kserve-6749469dd7-84h8b"] Jul 16 11:39:54.887869 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.887840 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc06635c-1e56-446d-9804-c64c54a83e42" path="/var/lib/kubelet/pods/bc06635c-1e56-446d-9804-c64c54a83e42/volumes" Jul 16 11:39:54.970260 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:54.970237 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:39:55.084018 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.083970 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-tls-certs\") pod \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " Jul 16 11:39:55.084222 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.084072 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-dshm\") pod \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " Jul 16 11:39:55.084222 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.084094 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-home\") pod \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " Jul 16 11:39:55.084222 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.084127 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr4rs\" (UniqueName: \"kubernetes.io/projected/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-kube-api-access-cr4rs\") pod \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " Jul 16 11:39:55.084222 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.084164 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-model-cache\") pod \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " Jul 16 11:39:55.084222 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.084210 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-tmp-dir\") pod \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\" (UID: \"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6\") " Jul 16 11:39:55.084781 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.084732 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "2652e4d4-d2c3-41f1-ab38-61e9895fa0a6" (UID: "2652e4d4-d2c3-41f1-ab38-61e9895fa0a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:39:55.084913 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.084773 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-home" (OuterVolumeSpecName: "home") pod "2652e4d4-d2c3-41f1-ab38-61e9895fa0a6" (UID: "2652e4d4-d2c3-41f1-ab38-61e9895fa0a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:39:55.084913 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.084791 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2652e4d4-d2c3-41f1-ab38-61e9895fa0a6" (UID: "2652e4d4-d2c3-41f1-ab38-61e9895fa0a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:39:55.088654 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.085131 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-home\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:39:55.088654 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.085154 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-model-cache\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:39:55.088654 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.085172 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-tmp-dir\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:39:55.088654 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.087085 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2652e4d4-d2c3-41f1-ab38-61e9895fa0a6" (UID: "2652e4d4-d2c3-41f1-ab38-61e9895fa0a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:39:55.088654 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.087922 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-dshm" (OuterVolumeSpecName: "dshm") pod "2652e4d4-d2c3-41f1-ab38-61e9895fa0a6" (UID: "2652e4d4-d2c3-41f1-ab38-61e9895fa0a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:39:55.093412 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.093385 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-kube-api-access-cr4rs" (OuterVolumeSpecName: "kube-api-access-cr4rs") pod "2652e4d4-d2c3-41f1-ab38-61e9895fa0a6" (UID: "2652e4d4-d2c3-41f1-ab38-61e9895fa0a6"). InnerVolumeSpecName "kube-api-access-cr4rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:39:55.186466 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.186371 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-dshm\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:39:55.186466 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.186413 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr4rs\" (UniqueName: \"kubernetes.io/projected/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-kube-api-access-cr4rs\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:39:55.186466 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.186425 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6-tls-certs\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:39:55.713103 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.713060 2580 generic.go:356] "Generic (PLEG): container finished" podID="2652e4d4-d2c3-41f1-ab38-61e9895fa0a6" containerID="88cc8e9697f864d266625289c10ada3225a5ee2f74aba8d458daa7eaf0353dc0" exitCode=0 Jul 16 11:39:55.713282 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.713111 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" Jul 16 11:39:55.713282 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.713124 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" event={"ID":"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6","Type":"ContainerDied","Data":"88cc8e9697f864d266625289c10ada3225a5ee2f74aba8d458daa7eaf0353dc0"} Jul 16 11:39:55.713282 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.713170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt" event={"ID":"2652e4d4-d2c3-41f1-ab38-61e9895fa0a6","Type":"ContainerDied","Data":"19e1753d25efde97e478f396fbda0b2ab5994487e64347dfe052e094840ddbd1"} Jul 16 11:39:55.713282 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.713191 2580 scope.go:117] "RemoveContainer" containerID="88cc8e9697f864d266625289c10ada3225a5ee2f74aba8d458daa7eaf0353dc0" Jul 16 11:39:55.722497 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.722449 2580 scope.go:117] "RemoveContainer" containerID="88cc8e9697f864d266625289c10ada3225a5ee2f74aba8d458daa7eaf0353dc0" Jul 16 11:39:55.722752 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:39:55.722724 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88cc8e9697f864d266625289c10ada3225a5ee2f74aba8d458daa7eaf0353dc0\": container with ID starting with 88cc8e9697f864d266625289c10ada3225a5ee2f74aba8d458daa7eaf0353dc0 not found: ID does not exist" containerID="88cc8e9697f864d266625289c10ada3225a5ee2f74aba8d458daa7eaf0353dc0" Jul 16 11:39:55.722837 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.722753 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cc8e9697f864d266625289c10ada3225a5ee2f74aba8d458daa7eaf0353dc0"} err="failed to get container status \"88cc8e9697f864d266625289c10ada3225a5ee2f74aba8d458daa7eaf0353dc0\": rpc error: code = NotFound desc = could not find container \"88cc8e9697f864d266625289c10ada3225a5ee2f74aba8d458daa7eaf0353dc0\": container with ID starting with 88cc8e9697f864d266625289c10ada3225a5ee2f74aba8d458daa7eaf0353dc0 not found: ID does not exist" Jul 16 11:39:55.734373 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.734349 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt"] Jul 16 11:39:55.738562 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:55.738537 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/auth-test-model-a47fa3-kserve-854c8c6487-bbptt"] Jul 16 11:39:56.885908 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:39:56.885872 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2652e4d4-d2c3-41f1-ab38-61e9895fa0a6" path="/var/lib/kubelet/pods/2652e4d4-d2c3-41f1-ab38-61e9895fa0a6/volumes" Jul 16 11:40:19.190374 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.190339 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd"] Jul 16 11:40:19.190763 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.190719 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2652e4d4-d2c3-41f1-ab38-61e9895fa0a6" containerName="main" Jul 16 11:40:19.190763 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.190733 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2652e4d4-d2c3-41f1-ab38-61e9895fa0a6" containerName="main" Jul 16 11:40:19.190763 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.190760 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc06635c-1e56-446d-9804-c64c54a83e42" containerName="main" Jul 16 11:40:19.190763 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.190765 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc06635c-1e56-446d-9804-c64c54a83e42" containerName="main" Jul 16 11:40:19.190900 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.190827 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc06635c-1e56-446d-9804-c64c54a83e42" containerName="main" Jul 16 11:40:19.190900 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.190836 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2652e4d4-d2c3-41f1-ab38-61e9895fa0a6" containerName="main" Jul 16 11:40:19.195376 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.195355 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.198117 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.198094 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a47fa3\"/\"kube-root-ca.crt\"" Jul 16 11:40:19.199268 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.199245 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a47fa3\"/\"sub-test-model-a47fa3-kserve-self-signed-certs\"" Jul 16 11:40:19.199362 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.199268 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a47fa3\"/\"openshift-service-ca.crt\"" Jul 16 11:40:19.199362 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.199345 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a47fa3\"/\"default-dockercfg-4vw9q\"" Jul 16 11:40:19.206452 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.206430 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd"] Jul 16 11:40:19.313030 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.312997 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-model-cache\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.313210 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.313049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-dshm\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.313210 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.313074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f07e9ada-698c-4bdc-87c3-a07085d267fd-tls-certs\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.313210 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.313125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5jhg\" (UniqueName: \"kubernetes.io/projected/f07e9ada-698c-4bdc-87c3-a07085d267fd-kube-api-access-m5jhg\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.313210 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.313185 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-tmp-dir\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.313210 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.313211 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-home\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.414550 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.414512 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-dshm\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.414746 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.414558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f07e9ada-698c-4bdc-87c3-a07085d267fd-tls-certs\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.414746 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.414614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5jhg\" (UniqueName: \"kubernetes.io/projected/f07e9ada-698c-4bdc-87c3-a07085d267fd-kube-api-access-m5jhg\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.414746 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.414676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-tmp-dir\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.414746 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.414711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-home\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.415022 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.414762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-model-cache\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.415082 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.415056 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-tmp-dir\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.415143 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.415123 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-home\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.415202 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.415161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-model-cache\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.416884 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.416857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-dshm\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.417127 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.417110 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f07e9ada-698c-4bdc-87c3-a07085d267fd-tls-certs\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.423512 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.423491 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5jhg\" (UniqueName: \"kubernetes.io/projected/f07e9ada-698c-4bdc-87c3-a07085d267fd-kube-api-access-m5jhg\") pod \"sub-test-model-a47fa3-kserve-6d589648c-jxgdd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.504754 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.504725 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:19.637939 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.637916 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd"] Jul 16 11:40:19.640437 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:40:19.640403 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf07e9ada_698c_4bdc_87c3_a07085d267fd.slice/crio-7fbefc237886165594f6ff436094d08a6443ca2a2177855be9d9348643d27385 WatchSource:0}: Error finding container 7fbefc237886165594f6ff436094d08a6443ca2a2177855be9d9348643d27385: Status 404 returned error can't find the container with id 7fbefc237886165594f6ff436094d08a6443ca2a2177855be9d9348643d27385 Jul 16 11:40:19.838258 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.838167 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" event={"ID":"f07e9ada-698c-4bdc-87c3-a07085d267fd","Type":"ContainerStarted","Data":"40c185d8caf1dd266c214517faa427b1057f5ac94c99b805c409f0a2e610c42f"} Jul 16 11:40:19.838258 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.838219 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" event={"ID":"f07e9ada-698c-4bdc-87c3-a07085d267fd","Type":"ContainerStarted","Data":"7fbefc237886165594f6ff436094d08a6443ca2a2177855be9d9348643d27385"} Jul 16 11:40:19.855117 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:19.855071 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" podStartSLOduration=0.855055454 podStartE2EDuration="855.055454ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:40:19.853684252 +0000 UTC m=+2191.687032938" watchObservedRunningTime="2026-07-16 11:40:19.855055454 +0000 UTC m=+2191.688404180" Jul 16 11:40:24.186183 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:24.186149 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:40:29.505721 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:29.505684 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:29.506113 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:29.505736 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:29.518393 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:29.518363 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:29.894968 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:29.894888 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:40:37.481705 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:37.481676 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:40:55.541204 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:55.541166 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd"] Jul 16 11:40:56.563441 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:56.563409 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd"] Jul 16 11:40:59.095758 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.095725 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk"] Jul 16 11:40:59.103509 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.103478 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.106068 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.106042 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8bae7d\"/\"rate-test-model-8bae7d-kserve-self-signed-certs\"" Jul 16 11:40:59.107015 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.106987 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk"] Jul 16 11:40:59.183396 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.183357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-model-cache\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.183602 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.183410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-home\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.183602 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.183462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-dshm\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.183602 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.183542 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-tmp-dir\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.183602 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.183597 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-tls-certs\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.183766 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.183615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kptgm\" (UniqueName: \"kubernetes.io/projected/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-kube-api-access-kptgm\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.284100 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.284072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-model-cache\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.284287 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.284115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-home\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.284287 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.284157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-dshm\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.284287 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.284222 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-tmp-dir\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.284496 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.284317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-tls-certs\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.284496 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.284372 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kptgm\" (UniqueName: \"kubernetes.io/projected/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-kube-api-access-kptgm\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.284616 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.284554 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-model-cache\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.284686 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.284662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-home\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.284830 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.284684 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-tmp-dir\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.286887 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.286866 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-dshm\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.287110 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.287091 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-tls-certs\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.294095 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.294068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kptgm\" (UniqueName: \"kubernetes.io/projected/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-kube-api-access-kptgm\") pod \"rate-test-model-8bae7d-kserve-564d74cb59-2lsxk\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.414764 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.414672 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:40:59.546033 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:40:59.546005 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk"] Jul 16 11:40:59.548827 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:40:59.548804 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc5bdfd2_8d8a_410d_b84d_30a1dce10a1e.slice/crio-80139f8836440c2d2916d458d31d7653611ce78c9f05779d647e7a2c873515fd WatchSource:0}: Error finding container 80139f8836440c2d2916d458d31d7653611ce78c9f05779d647e7a2c873515fd: Status 404 returned error can't find the container with id 80139f8836440c2d2916d458d31d7653611ce78c9f05779d647e7a2c873515fd Jul 16 11:41:00.023937 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:00.023898 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" event={"ID":"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e","Type":"ContainerStarted","Data":"3993d9e959c4f833a5b2e14b621106875a260120ee439d77eff8349cc2b9474b"} Jul 16 11:41:00.024110 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:00.023949 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" event={"ID":"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e","Type":"ContainerStarted","Data":"80139f8836440c2d2916d458d31d7653611ce78c9f05779d647e7a2c873515fd"} Jul 16 11:41:00.045264 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:00.045212 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" podStartSLOduration=1.045179267 podStartE2EDuration="1.045179267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:41:00.044499901 +0000 UTC m=+2231.877848585" watchObservedRunningTime="2026-07-16 11:41:00.045179267 +0000 UTC m=+2231.878527948" Jul 16 11:41:03.186395 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:03.186359 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:41:09.415026 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:09.414990 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:41:09.415514 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:09.415132 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:41:09.427656 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:09.427626 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:41:10.078985 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.078955 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:41:10.562165 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.562114 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" podUID="e23d07b4-95be-4eb3-92ae-b2a03769ec93" containerName="main" containerID="cri-o://033ffe6435062acc49225ef480dfaa0b96cf8184c7908030e8de6ade1cca2c10" gracePeriod=45 Jul 16 11:41:10.809635 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.809604 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:41:10.886443 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.886363 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-tmp-dir\") pod \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " Jul 16 11:41:10.886443 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.886401 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-dshm\") pod \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " Jul 16 11:41:10.886650 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.886447 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rhv6\" (UniqueName: \"kubernetes.io/projected/e23d07b4-95be-4eb3-92ae-b2a03769ec93-kube-api-access-8rhv6\") pod \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " Jul 16 11:41:10.886650 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.886472 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-model-cache\") pod \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " Jul 16 11:41:10.886650 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.886530 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23d07b4-95be-4eb3-92ae-b2a03769ec93-tls-certs\") pod \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " Jul 16 11:41:10.886650 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.886645 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-home\") pod \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\" (UID: \"e23d07b4-95be-4eb3-92ae-b2a03769ec93\") " Jul 16 11:41:10.886860 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.886667 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e23d07b4-95be-4eb3-92ae-b2a03769ec93" (UID: "e23d07b4-95be-4eb3-92ae-b2a03769ec93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:41:10.886860 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.886818 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-model-cache" (OuterVolumeSpecName: "model-cache") pod "e23d07b4-95be-4eb3-92ae-b2a03769ec93" (UID: "e23d07b4-95be-4eb3-92ae-b2a03769ec93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:41:10.886975 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.886906 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-model-cache\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:41:10.886975 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.886932 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-tmp-dir\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:41:10.887058 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.886973 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-home" (OuterVolumeSpecName: "home") pod "e23d07b4-95be-4eb3-92ae-b2a03769ec93" (UID: "e23d07b4-95be-4eb3-92ae-b2a03769ec93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:41:10.888639 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.888615 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23d07b4-95be-4eb3-92ae-b2a03769ec93-kube-api-access-8rhv6" (OuterVolumeSpecName: "kube-api-access-8rhv6") pod "e23d07b4-95be-4eb3-92ae-b2a03769ec93" (UID: "e23d07b4-95be-4eb3-92ae-b2a03769ec93"). InnerVolumeSpecName "kube-api-access-8rhv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:41:10.888743 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.888683 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23d07b4-95be-4eb3-92ae-b2a03769ec93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e23d07b4-95be-4eb3-92ae-b2a03769ec93" (UID: "e23d07b4-95be-4eb3-92ae-b2a03769ec93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:41:10.888837 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.888816 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-dshm" (OuterVolumeSpecName: "dshm") pod "e23d07b4-95be-4eb3-92ae-b2a03769ec93" (UID: "e23d07b4-95be-4eb3-92ae-b2a03769ec93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:41:10.987451 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.987412 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-dshm\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:41:10.987451 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.987440 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rhv6\" (UniqueName: \"kubernetes.io/projected/e23d07b4-95be-4eb3-92ae-b2a03769ec93-kube-api-access-8rhv6\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:41:10.987451 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.987456 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23d07b4-95be-4eb3-92ae-b2a03769ec93-tls-certs\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:41:10.987711 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:10.987467 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23d07b4-95be-4eb3-92ae-b2a03769ec93-home\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:41:11.074012 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.073978 2580 generic.go:356] "Generic (PLEG): container finished" podID="e23d07b4-95be-4eb3-92ae-b2a03769ec93" containerID="033ffe6435062acc49225ef480dfaa0b96cf8184c7908030e8de6ade1cca2c10" exitCode=0 Jul 16 11:41:11.074190 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.074024 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" Jul 16 11:41:11.074190 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.074065 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" event={"ID":"e23d07b4-95be-4eb3-92ae-b2a03769ec93","Type":"ContainerDied","Data":"033ffe6435062acc49225ef480dfaa0b96cf8184c7908030e8de6ade1cca2c10"} Jul 16 11:41:11.074190 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.074105 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd" event={"ID":"e23d07b4-95be-4eb3-92ae-b2a03769ec93","Type":"ContainerDied","Data":"46ee8503790f2c5c9714a6f0b9836b8c4e8545e346557b87e152300a3a79073d"} Jul 16 11:41:11.074190 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.074125 2580 scope.go:117] "RemoveContainer" containerID="033ffe6435062acc49225ef480dfaa0b96cf8184c7908030e8de6ade1cca2c10" Jul 16 11:41:11.083203 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.083186 2580 scope.go:117] "RemoveContainer" containerID="033ffe6435062acc49225ef480dfaa0b96cf8184c7908030e8de6ade1cca2c10" Jul 16 11:41:11.083468 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:41:11.083450 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033ffe6435062acc49225ef480dfaa0b96cf8184c7908030e8de6ade1cca2c10\": container with ID starting with 033ffe6435062acc49225ef480dfaa0b96cf8184c7908030e8de6ade1cca2c10 not found: ID does not exist" containerID="033ffe6435062acc49225ef480dfaa0b96cf8184c7908030e8de6ade1cca2c10" Jul 16 11:41:11.083545 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.083475 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033ffe6435062acc49225ef480dfaa0b96cf8184c7908030e8de6ade1cca2c10"} err="failed to get container status \"033ffe6435062acc49225ef480dfaa0b96cf8184c7908030e8de6ade1cca2c10\": rpc error: code = NotFound desc = could not find container \"033ffe6435062acc49225ef480dfaa0b96cf8184c7908030e8de6ade1cca2c10\": container with ID starting with 033ffe6435062acc49225ef480dfaa0b96cf8184c7908030e8de6ade1cca2c10 not found: ID does not exist" Jul 16 11:41:11.096335 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.096310 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd"] Jul 16 11:41:11.100019 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.100000 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/sub-test-model-8bae7d-kserve-58f4f5d9fc-2b8xd"] Jul 16 11:41:11.585284 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.585214 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" podUID="f07e9ada-698c-4bdc-87c3-a07085d267fd" containerName="main" containerID="cri-o://40c185d8caf1dd266c214517faa427b1057f5ac94c99b805c409f0a2e610c42f" gracePeriod=45 Jul 16 11:41:11.828260 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.828237 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:41:11.901386 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.901311 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f07e9ada-698c-4bdc-87c3-a07085d267fd-tls-certs\") pod \"f07e9ada-698c-4bdc-87c3-a07085d267fd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " Jul 16 11:41:11.901386 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.901347 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-tmp-dir\") pod \"f07e9ada-698c-4bdc-87c3-a07085d267fd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " Jul 16 11:41:11.901617 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.901393 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-home\") pod \"f07e9ada-698c-4bdc-87c3-a07085d267fd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " Jul 16 11:41:11.901617 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.901529 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-model-cache\") pod \"f07e9ada-698c-4bdc-87c3-a07085d267fd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " Jul 16 11:41:11.901617 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.901563 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5jhg\" (UniqueName: \"kubernetes.io/projected/f07e9ada-698c-4bdc-87c3-a07085d267fd-kube-api-access-m5jhg\") pod \"f07e9ada-698c-4bdc-87c3-a07085d267fd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " Jul 16 11:41:11.901617 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.901617 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-dshm\") pod \"f07e9ada-698c-4bdc-87c3-a07085d267fd\" (UID: \"f07e9ada-698c-4bdc-87c3-a07085d267fd\") " Jul 16 11:41:11.901827 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.901675 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f07e9ada-698c-4bdc-87c3-a07085d267fd" (UID: "f07e9ada-698c-4bdc-87c3-a07085d267fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:41:11.901827 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.901683 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-home" (OuterVolumeSpecName: "home") pod "f07e9ada-698c-4bdc-87c3-a07085d267fd" (UID: "f07e9ada-698c-4bdc-87c3-a07085d267fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:41:11.901827 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.901802 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "f07e9ada-698c-4bdc-87c3-a07085d267fd" (UID: "f07e9ada-698c-4bdc-87c3-a07085d267fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:41:11.901983 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.901966 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-home\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:41:11.902049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.901984 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-model-cache\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:41:11.902049 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.901995 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-tmp-dir\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:41:11.903459 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.903437 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f07e9ada-698c-4bdc-87c3-a07085d267fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f07e9ada-698c-4bdc-87c3-a07085d267fd" (UID: "f07e9ada-698c-4bdc-87c3-a07085d267fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:41:11.903614 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.903592 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-dshm" (OuterVolumeSpecName: "dshm") pod "f07e9ada-698c-4bdc-87c3-a07085d267fd" (UID: "f07e9ada-698c-4bdc-87c3-a07085d267fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:41:11.903757 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:11.903739 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07e9ada-698c-4bdc-87c3-a07085d267fd-kube-api-access-m5jhg" (OuterVolumeSpecName: "kube-api-access-m5jhg") pod "f07e9ada-698c-4bdc-87c3-a07085d267fd" (UID: "f07e9ada-698c-4bdc-87c3-a07085d267fd"). InnerVolumeSpecName "kube-api-access-m5jhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:41:12.003172 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:12.003128 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5jhg\" (UniqueName: \"kubernetes.io/projected/f07e9ada-698c-4bdc-87c3-a07085d267fd-kube-api-access-m5jhg\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:41:12.003172 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:12.003160 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f07e9ada-698c-4bdc-87c3-a07085d267fd-dshm\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:41:12.003172 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:12.003172 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f07e9ada-698c-4bdc-87c3-a07085d267fd-tls-certs\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:41:12.080980 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:12.080950 2580 generic.go:356] "Generic (PLEG): container finished" podID="f07e9ada-698c-4bdc-87c3-a07085d267fd" containerID="40c185d8caf1dd266c214517faa427b1057f5ac94c99b805c409f0a2e610c42f" exitCode=0 Jul 16 11:41:12.081159 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:12.080997 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" Jul 16 11:41:12.081159 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:12.081031 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" event={"ID":"f07e9ada-698c-4bdc-87c3-a07085d267fd","Type":"ContainerDied","Data":"40c185d8caf1dd266c214517faa427b1057f5ac94c99b805c409f0a2e610c42f"} Jul 16 11:41:12.081159 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:12.081073 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd" event={"ID":"f07e9ada-698c-4bdc-87c3-a07085d267fd","Type":"ContainerDied","Data":"7fbefc237886165594f6ff436094d08a6443ca2a2177855be9d9348643d27385"} Jul 16 11:41:12.081159 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:12.081096 2580 scope.go:117] "RemoveContainer" containerID="40c185d8caf1dd266c214517faa427b1057f5ac94c99b805c409f0a2e610c42f" Jul 16 11:41:12.089940 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:12.089924 2580 scope.go:117] "RemoveContainer" containerID="40c185d8caf1dd266c214517faa427b1057f5ac94c99b805c409f0a2e610c42f" Jul 16 11:41:12.090226 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:41:12.090205 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c185d8caf1dd266c214517faa427b1057f5ac94c99b805c409f0a2e610c42f\": container with ID starting with 40c185d8caf1dd266c214517faa427b1057f5ac94c99b805c409f0a2e610c42f not found: ID does not exist" containerID="40c185d8caf1dd266c214517faa427b1057f5ac94c99b805c409f0a2e610c42f" Jul 16 11:41:12.090299 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:12.090233 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c185d8caf1dd266c214517faa427b1057f5ac94c99b805c409f0a2e610c42f"} err="failed to get container status \"40c185d8caf1dd266c214517faa427b1057f5ac94c99b805c409f0a2e610c42f\": rpc error: code = NotFound desc = could not find container \"40c185d8caf1dd266c214517faa427b1057f5ac94c99b805c409f0a2e610c42f\": container with ID starting with 40c185d8caf1dd266c214517faa427b1057f5ac94c99b805c409f0a2e610c42f not found: ID does not exist" Jul 16 11:41:12.103226 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:12.103203 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd"] Jul 16 11:41:12.107418 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:12.107398 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/sub-test-model-a47fa3-kserve-6d589648c-jxgdd"] Jul 16 11:41:12.886091 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:12.886052 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e23d07b4-95be-4eb3-92ae-b2a03769ec93" path="/var/lib/kubelet/pods/e23d07b4-95be-4eb3-92ae-b2a03769ec93/volumes" Jul 16 11:41:12.886663 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:12.886643 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f07e9ada-698c-4bdc-87c3-a07085d267fd" path="/var/lib/kubelet/pods/f07e9ada-698c-4bdc-87c3-a07085d267fd/volumes" Jul 16 11:41:14.678720 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.678686 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd"] Jul 16 11:41:14.679080 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.679046 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e23d07b4-95be-4eb3-92ae-b2a03769ec93" containerName="main" Jul 16 11:41:14.679080 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.679059 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e23d07b4-95be-4eb3-92ae-b2a03769ec93" containerName="main" Jul 16 11:41:14.679166 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.679085 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f07e9ada-698c-4bdc-87c3-a07085d267fd" containerName="main" Jul 16 11:41:14.679166 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.679091 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f07e9ada-698c-4bdc-87c3-a07085d267fd" containerName="main" Jul 16 11:41:14.679166 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.679138 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f07e9ada-698c-4bdc-87c3-a07085d267fd" containerName="main" Jul 16 11:41:14.679166 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.679148 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e23d07b4-95be-4eb3-92ae-b2a03769ec93" containerName="main" Jul 16 11:41:14.682934 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.682913 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.686975 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.686947 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a47fa3\"/\"kube-root-ca.crt\"" Jul 16 11:41:14.687096 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.686947 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a47fa3\"/\"default-dockercfg-4vw9q\"" Jul 16 11:41:14.687096 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.686949 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a47fa3\"/\"openshift-service-ca.crt\"" Jul 16 11:41:14.687096 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.687063 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a47fa3\"/\"rate-test-model-a47fa3-kserve-self-signed-certs\"" Jul 16 11:41:14.692542 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.692524 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd"] Jul 16 11:41:14.829863 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.829830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f05eec47-32db-46bd-af76-7a8e4d995b89-tls-certs\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.830060 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.829877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5d6v\" (UniqueName: \"kubernetes.io/projected/f05eec47-32db-46bd-af76-7a8e4d995b89-kube-api-access-f5d6v\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.830060 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.829952 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-dshm\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.830060 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.830008 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-tmp-dir\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.830060 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.830057 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-home\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.830228 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.830081 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-model-cache\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.931339 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.931247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-dshm\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.931339 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.931282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-tmp-dir\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.931339 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.931307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-home\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.931641 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.931423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-model-cache\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.931641 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.931518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f05eec47-32db-46bd-af76-7a8e4d995b89-tls-certs\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.931641 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.931565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5d6v\" (UniqueName: \"kubernetes.io/projected/f05eec47-32db-46bd-af76-7a8e4d995b89-kube-api-access-f5d6v\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.931793 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.931689 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-tmp-dir\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.931793 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.931782 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-home\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.931868 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.931853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-model-cache\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.933705 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.933677 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-dshm\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.933870 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.933851 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f05eec47-32db-46bd-af76-7a8e4d995b89-tls-certs\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.939749 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.939722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5d6v\" (UniqueName: \"kubernetes.io/projected/f05eec47-32db-46bd-af76-7a8e4d995b89-kube-api-access-f5d6v\") pod \"rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:14.993142 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:14.993113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:15.127351 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:15.127327 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd"] Jul 16 11:41:15.130058 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:41:15.130029 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf05eec47_32db_46bd_af76_7a8e4d995b89.slice/crio-7645e3814b6209bc73c6bddb3b7a232e2a480aa2008d5e57c3431740e660531e WatchSource:0}: Error finding container 7645e3814b6209bc73c6bddb3b7a232e2a480aa2008d5e57c3431740e660531e: Status 404 returned error can't find the container with id 7645e3814b6209bc73c6bddb3b7a232e2a480aa2008d5e57c3431740e660531e Jul 16 11:41:16.106234 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:16.106194 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" event={"ID":"f05eec47-32db-46bd-af76-7a8e4d995b89","Type":"ContainerStarted","Data":"c487ea42886c77ad5882316e2f583da87d724b4e6e3fdf1a242f72260eb942dd"} Jul 16 11:41:16.106647 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:16.106245 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" event={"ID":"f05eec47-32db-46bd-af76-7a8e4d995b89","Type":"ContainerStarted","Data":"7645e3814b6209bc73c6bddb3b7a232e2a480aa2008d5e57c3431740e660531e"} Jul 16 11:41:16.123650 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:16.123602 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" podStartSLOduration=2.123587178 podStartE2EDuration="2.123587178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:41:16.122658028 +0000 UTC m=+2247.956006711" watchObservedRunningTime="2026-07-16 11:41:16.123587178 +0000 UTC m=+2247.956935851" Jul 16 11:41:21.087861 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:21.087826 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:41:24.993697 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:24.993661 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:24.994062 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:24.993723 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:25.006109 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:25.006083 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:25.159810 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:25.159784 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:41:42.092000 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:42.091966 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:41:57.581539 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:41:57.581506 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:42:04.971362 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:04.971326 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk"] Jul 16 11:42:05.912493 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:05.912461 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd"] Jul 16 11:42:06.923536 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:06.923504 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7442-140371682750768"] Jul 16 11:42:06.927072 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:06.927056 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371682750768" Jul 16 11:42:06.929568 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:06.929542 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zb2bt\"" Jul 16 11:42:06.933829 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:06.933799 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140371682750768"] Jul 16 11:42:07.111703 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:07.111668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9pb2\" (UniqueName: \"kubernetes.io/projected/e47ef3de-820c-4c64-a9bb-84f4fe13c413-kube-api-access-h9pb2\") pod \"test-curl-7442-140371682750768\" (UID: \"e47ef3de-820c-4c64-a9bb-84f4fe13c413\") " pod="opendatahub/test-curl-7442-140371682750768" Jul 16 11:42:07.212963 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:07.212866 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9pb2\" (UniqueName: \"kubernetes.io/projected/e47ef3de-820c-4c64-a9bb-84f4fe13c413-kube-api-access-h9pb2\") pod \"test-curl-7442-140371682750768\" (UID: \"e47ef3de-820c-4c64-a9bb-84f4fe13c413\") " pod="opendatahub/test-curl-7442-140371682750768" Jul 16 11:42:07.222153 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:07.222118 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9pb2\" (UniqueName: \"kubernetes.io/projected/e47ef3de-820c-4c64-a9bb-84f4fe13c413-kube-api-access-h9pb2\") pod \"test-curl-7442-140371682750768\" (UID: \"e47ef3de-820c-4c64-a9bb-84f4fe13c413\") " pod="opendatahub/test-curl-7442-140371682750768" Jul 16 11:42:07.237030 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:07.237002 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371682750768" Jul 16 11:42:07.364622 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:07.364563 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140371682750768"] Jul 16 11:42:07.367346 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:42:07.367318 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode47ef3de_820c_4c64_a9bb_84f4fe13c413.slice/crio-9f031d798a9790e9a027407534ed63c36f85c5900681b75a243f2389aad600e8 WatchSource:0}: Error finding container 9f031d798a9790e9a027407534ed63c36f85c5900681b75a243f2389aad600e8: Status 404 returned error can't find the container with id 9f031d798a9790e9a027407534ed63c36f85c5900681b75a243f2389aad600e8 Jul 16 11:42:07.368626 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:07.368607 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 11:42:08.353871 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:08.353781 2580 generic.go:356] "Generic (PLEG): container finished" podID="e47ef3de-820c-4c64-a9bb-84f4fe13c413" containerID="c79ecf6e0f52ae512ed0ff3acf5e45b9b49f2737237af51692b58028112e9105" exitCode=0 Jul 16 11:42:08.353871 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:08.353855 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140371682750768" event={"ID":"e47ef3de-820c-4c64-a9bb-84f4fe13c413","Type":"ContainerDied","Data":"c79ecf6e0f52ae512ed0ff3acf5e45b9b49f2737237af51692b58028112e9105"} Jul 16 11:42:08.354364 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:08.353888 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140371682750768" event={"ID":"e47ef3de-820c-4c64-a9bb-84f4fe13c413","Type":"ContainerStarted","Data":"9f031d798a9790e9a027407534ed63c36f85c5900681b75a243f2389aad600e8"} Jul 16 11:42:09.490594 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:09.490554 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371682750768" Jul 16 11:42:09.547710 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:09.547685 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7442-140371682750768_e47ef3de-820c-4c64-a9bb-84f4fe13c413/test-curl-7442-140371682750768/0.log" Jul 16 11:42:09.583620 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:09.583591 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7442-140371682750768"] Jul 16 11:42:09.590504 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:09.590479 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7442-140371682750768"] Jul 16 11:42:09.632432 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:09.632360 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9pb2\" (UniqueName: \"kubernetes.io/projected/e47ef3de-820c-4c64-a9bb-84f4fe13c413-kube-api-access-h9pb2\") pod \"e47ef3de-820c-4c64-a9bb-84f4fe13c413\" (UID: \"e47ef3de-820c-4c64-a9bb-84f4fe13c413\") " Jul 16 11:42:09.634618 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:09.634592 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47ef3de-820c-4c64-a9bb-84f4fe13c413-kube-api-access-h9pb2" (OuterVolumeSpecName: "kube-api-access-h9pb2") pod "e47ef3de-820c-4c64-a9bb-84f4fe13c413" (UID: "e47ef3de-820c-4c64-a9bb-84f4fe13c413"). InnerVolumeSpecName "kube-api-access-h9pb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:42:09.733167 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:09.733116 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9pb2\" (UniqueName: \"kubernetes.io/projected/e47ef3de-820c-4c64-a9bb-84f4fe13c413-kube-api-access-h9pb2\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:10.364369 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:10.364337 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f031d798a9790e9a027407534ed63c36f85c5900681b75a243f2389aad600e8" Jul 16 11:42:10.364369 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:10.364352 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371682750768" Jul 16 11:42:10.374460 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:42:10.374430 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7442-140371682750768\" is forbidden: User \"system:node:ip-10-0-142-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-15.ec2.internal' and this object" podUID="e47ef3de-820c-4c64-a9bb-84f4fe13c413" pod="opendatahub/test-curl-7442-140371682750768" Jul 16 11:42:10.886238 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:10.886204 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e47ef3de-820c-4c64-a9bb-84f4fe13c413" path="/var/lib/kubelet/pods/e47ef3de-820c-4c64-a9bb-84f4fe13c413/volumes" Jul 16 11:42:11.995400 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:11.995367 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:42:12.825856 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:12.825819 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7442-140371684679232"] Jul 16 11:42:12.826181 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:12.826168 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e47ef3de-820c-4c64-a9bb-84f4fe13c413" containerName="test-curl-7442-140371682750768" Jul 16 11:42:12.826223 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:12.826182 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e47ef3de-820c-4c64-a9bb-84f4fe13c413" containerName="test-curl-7442-140371682750768" Jul 16 11:42:12.826267 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:12.826258 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e47ef3de-820c-4c64-a9bb-84f4fe13c413" containerName="test-curl-7442-140371682750768" Jul 16 11:42:12.829362 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:12.829344 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371684679232" Jul 16 11:42:12.832067 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:12.832050 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zb2bt\"" Jul 16 11:42:12.835852 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:12.835827 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140371684679232"] Jul 16 11:42:12.861676 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:12.861646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn6rd\" (UniqueName: \"kubernetes.io/projected/b92b4914-e0c5-4ac2-a4ac-705952c257e1-kube-api-access-qn6rd\") pod \"test-curl-7442-140371684679232\" (UID: \"b92b4914-e0c5-4ac2-a4ac-705952c257e1\") " pod="opendatahub/test-curl-7442-140371684679232" Jul 16 11:42:12.962532 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:12.962500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn6rd\" (UniqueName: \"kubernetes.io/projected/b92b4914-e0c5-4ac2-a4ac-705952c257e1-kube-api-access-qn6rd\") pod \"test-curl-7442-140371684679232\" (UID: \"b92b4914-e0c5-4ac2-a4ac-705952c257e1\") " pod="opendatahub/test-curl-7442-140371684679232" Jul 16 11:42:12.971189 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:12.971156 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn6rd\" (UniqueName: \"kubernetes.io/projected/b92b4914-e0c5-4ac2-a4ac-705952c257e1-kube-api-access-qn6rd\") pod \"test-curl-7442-140371684679232\" (UID: \"b92b4914-e0c5-4ac2-a4ac-705952c257e1\") " pod="opendatahub/test-curl-7442-140371684679232" Jul 16 11:42:13.138871 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:13.138787 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371684679232" Jul 16 11:42:13.267389 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:13.267359 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140371684679232"] Jul 16 11:42:13.269425 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:42:13.269387 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb92b4914_e0c5_4ac2_a4ac_705952c257e1.slice/crio-ea6e136c319006ad1d859702a92d8cb492c188a5a4e5dca0c82e41fdf1e4443a WatchSource:0}: Error finding container ea6e136c319006ad1d859702a92d8cb492c188a5a4e5dca0c82e41fdf1e4443a: Status 404 returned error can't find the container with id ea6e136c319006ad1d859702a92d8cb492c188a5a4e5dca0c82e41fdf1e4443a Jul 16 11:42:13.380188 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:13.380159 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140371684679232" event={"ID":"b92b4914-e0c5-4ac2-a4ac-705952c257e1","Type":"ContainerStarted","Data":"ea6e136c319006ad1d859702a92d8cb492c188a5a4e5dca0c82e41fdf1e4443a"} Jul 16 11:42:14.386038 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:14.386008 2580 generic.go:356] "Generic (PLEG): container finished" podID="b92b4914-e0c5-4ac2-a4ac-705952c257e1" containerID="c32c1416844760cea368840e8b58b4c506514b7feb08bd2f717c2a8593f2bec8" exitCode=0 Jul 16 11:42:14.386454 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:14.386098 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140371684679232" event={"ID":"b92b4914-e0c5-4ac2-a4ac-705952c257e1","Type":"ContainerDied","Data":"c32c1416844760cea368840e8b58b4c506514b7feb08bd2f717c2a8593f2bec8"} Jul 16 11:42:15.529790 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:15.529764 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371684679232" Jul 16 11:42:15.551601 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:15.551555 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7442-140371684679232_b92b4914-e0c5-4ac2-a4ac-705952c257e1/test-curl-7442-140371684679232/0.log" Jul 16 11:42:15.578243 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:15.578213 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7442-140371684679232"] Jul 16 11:42:15.582590 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:15.582556 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7442-140371684679232"] Jul 16 11:42:15.584996 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:15.584980 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn6rd\" (UniqueName: \"kubernetes.io/projected/b92b4914-e0c5-4ac2-a4ac-705952c257e1-kube-api-access-qn6rd\") pod \"b92b4914-e0c5-4ac2-a4ac-705952c257e1\" (UID: \"b92b4914-e0c5-4ac2-a4ac-705952c257e1\") " Jul 16 11:42:15.586962 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:15.586939 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92b4914-e0c5-4ac2-a4ac-705952c257e1-kube-api-access-qn6rd" (OuterVolumeSpecName: "kube-api-access-qn6rd") pod "b92b4914-e0c5-4ac2-a4ac-705952c257e1" (UID: "b92b4914-e0c5-4ac2-a4ac-705952c257e1"). InnerVolumeSpecName "kube-api-access-qn6rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:42:15.685713 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:15.685615 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn6rd\" (UniqueName: \"kubernetes.io/projected/b92b4914-e0c5-4ac2-a4ac-705952c257e1-kube-api-access-qn6rd\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:16.402773 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:16.402742 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea6e136c319006ad1d859702a92d8cb492c188a5a4e5dca0c82e41fdf1e4443a" Jul 16 11:42:16.402773 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:16.402762 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371684679232" Jul 16 11:42:16.412833 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:42:16.412802 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7442-140371684679232\" is forbidden: User \"system:node:ip-10-0-142-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-15.ec2.internal' and this object" podUID="b92b4914-e0c5-4ac2-a4ac-705952c257e1" pod="opendatahub/test-curl-7442-140371684679232" Jul 16 11:42:16.886126 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:16.886088 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b92b4914-e0c5-4ac2-a4ac-705952c257e1" path="/var/lib/kubelet/pods/b92b4914-e0c5-4ac2-a4ac-705952c257e1/volumes" Jul 16 11:42:18.830415 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:18.830384 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7442-140371679974000"] Jul 16 11:42:18.830806 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:18.830762 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b92b4914-e0c5-4ac2-a4ac-705952c257e1" containerName="test-curl-7442-140371684679232" Jul 16 11:42:18.830806 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:18.830777 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b92b4914-e0c5-4ac2-a4ac-705952c257e1" containerName="test-curl-7442-140371684679232" Jul 16 11:42:18.830886 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:18.830852 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b92b4914-e0c5-4ac2-a4ac-705952c257e1" containerName="test-curl-7442-140371684679232" Jul 16 11:42:18.833995 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:18.833978 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371679974000" Jul 16 11:42:18.836516 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:18.836493 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zb2bt\"" Jul 16 11:42:18.841269 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:18.841024 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140371679974000"] Jul 16 11:42:18.917497 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:18.917464 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kllv\" (UniqueName: \"kubernetes.io/projected/e1b40d78-2230-4333-be66-51dccc570332-kube-api-access-5kllv\") pod \"test-curl-7442-140371679974000\" (UID: \"e1b40d78-2230-4333-be66-51dccc570332\") " pod="opendatahub/test-curl-7442-140371679974000" Jul 16 11:42:19.018676 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:19.018637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kllv\" (UniqueName: \"kubernetes.io/projected/e1b40d78-2230-4333-be66-51dccc570332-kube-api-access-5kllv\") pod \"test-curl-7442-140371679974000\" (UID: \"e1b40d78-2230-4333-be66-51dccc570332\") " pod="opendatahub/test-curl-7442-140371679974000" Jul 16 11:42:19.028741 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:19.028716 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kllv\" (UniqueName: \"kubernetes.io/projected/e1b40d78-2230-4333-be66-51dccc570332-kube-api-access-5kllv\") pod \"test-curl-7442-140371679974000\" (UID: \"e1b40d78-2230-4333-be66-51dccc570332\") " pod="opendatahub/test-curl-7442-140371679974000" Jul 16 11:42:19.143470 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:19.143384 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371679974000" Jul 16 11:42:19.282863 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:19.282820 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140371679974000"] Jul 16 11:42:19.285740 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:42:19.285715 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1b40d78_2230_4333_be66_51dccc570332.slice/crio-203646370bbc0763b329bce95fb7baf45b235a0ee0eb4ec9060c5f11dd6c6c19 WatchSource:0}: Error finding container 203646370bbc0763b329bce95fb7baf45b235a0ee0eb4ec9060c5f11dd6c6c19: Status 404 returned error can't find the container with id 203646370bbc0763b329bce95fb7baf45b235a0ee0eb4ec9060c5f11dd6c6c19 Jul 16 11:42:19.417135 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:19.417046 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140371679974000" event={"ID":"e1b40d78-2230-4333-be66-51dccc570332","Type":"ContainerStarted","Data":"203646370bbc0763b329bce95fb7baf45b235a0ee0eb4ec9060c5f11dd6c6c19"} Jul 16 11:42:19.991380 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:19.991280 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" podUID="fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e" containerName="main" containerID="cri-o://3993d9e959c4f833a5b2e14b621106875a260120ee439d77eff8349cc2b9474b" gracePeriod=45 Jul 16 11:42:20.068214 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.068174 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" podUID="fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": dial tcp 10.134.0.34:8000: connect: connection refused" Jul 16 11:42:20.242650 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.242562 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:42:20.333146 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.333114 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-model-cache\") pod \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " Jul 16 11:42:20.333346 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.333154 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-home\") pod \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " Jul 16 11:42:20.333346 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.333211 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-tmp-dir\") pod \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " Jul 16 11:42:20.333346 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.333241 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-dshm\") pod \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " Jul 16 11:42:20.333944 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.333358 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-tls-certs\") pod \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " Jul 16 11:42:20.333944 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.333388 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kptgm\" (UniqueName: \"kubernetes.io/projected/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-kube-api-access-kptgm\") pod \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\" (UID: \"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e\") " Jul 16 11:42:20.333944 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.333430 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e" (UID: "fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:42:20.333944 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.333493 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e" (UID: "fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:42:20.333944 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.333516 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-home" (OuterVolumeSpecName: "home") pod "fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e" (UID: "fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:42:20.333944 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.333705 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-model-cache\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:20.333944 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.333719 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-home\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:20.333944 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.333727 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-tmp-dir\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:20.335426 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.335396 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-dshm" (OuterVolumeSpecName: "dshm") pod "fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e" (UID: "fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:42:20.335542 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.335430 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e" (UID: "fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:42:20.335542 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.335483 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-kube-api-access-kptgm" (OuterVolumeSpecName: "kube-api-access-kptgm") pod "fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e" (UID: "fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e"). InnerVolumeSpecName "kube-api-access-kptgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:42:20.429946 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.429904 2580 generic.go:356] "Generic (PLEG): container finished" podID="e1b40d78-2230-4333-be66-51dccc570332" containerID="bcdafff09ed5329cc0b75c334bc76e5817012fff9ab591078deecd368c52f9ea" exitCode=0 Jul 16 11:42:20.430142 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.429948 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140371679974000" event={"ID":"e1b40d78-2230-4333-be66-51dccc570332","Type":"ContainerDied","Data":"bcdafff09ed5329cc0b75c334bc76e5817012fff9ab591078deecd368c52f9ea"} Jul 16 11:42:20.431659 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.431633 2580 generic.go:356] "Generic (PLEG): container finished" podID="fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e" containerID="3993d9e959c4f833a5b2e14b621106875a260120ee439d77eff8349cc2b9474b" exitCode=0 Jul 16 11:42:20.431809 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.431675 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" Jul 16 11:42:20.431809 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.431685 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" event={"ID":"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e","Type":"ContainerDied","Data":"3993d9e959c4f833a5b2e14b621106875a260120ee439d77eff8349cc2b9474b"} Jul 16 11:42:20.431809 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.431716 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk" event={"ID":"fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e","Type":"ContainerDied","Data":"80139f8836440c2d2916d458d31d7653611ce78c9f05779d647e7a2c873515fd"} Jul 16 11:42:20.431809 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.431738 2580 scope.go:117] "RemoveContainer" containerID="3993d9e959c4f833a5b2e14b621106875a260120ee439d77eff8349cc2b9474b" Jul 16 11:42:20.434249 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.434220 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-tls-certs\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:20.434362 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.434252 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kptgm\" (UniqueName: \"kubernetes.io/projected/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-kube-api-access-kptgm\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:20.434362 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.434266 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e-dshm\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:20.443427 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.443404 2580 scope.go:117] "RemoveContainer" containerID="3993d9e959c4f833a5b2e14b621106875a260120ee439d77eff8349cc2b9474b" Jul 16 11:42:20.444166 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:42:20.444138 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3993d9e959c4f833a5b2e14b621106875a260120ee439d77eff8349cc2b9474b\": container with ID starting with 3993d9e959c4f833a5b2e14b621106875a260120ee439d77eff8349cc2b9474b not found: ID does not exist" containerID="3993d9e959c4f833a5b2e14b621106875a260120ee439d77eff8349cc2b9474b" Jul 16 11:42:20.444258 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.444175 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3993d9e959c4f833a5b2e14b621106875a260120ee439d77eff8349cc2b9474b"} err="failed to get container status \"3993d9e959c4f833a5b2e14b621106875a260120ee439d77eff8349cc2b9474b\": rpc error: code = NotFound desc = could not find container \"3993d9e959c4f833a5b2e14b621106875a260120ee439d77eff8349cc2b9474b\": container with ID starting with 3993d9e959c4f833a5b2e14b621106875a260120ee439d77eff8349cc2b9474b not found: ID does not exist" Jul 16 11:42:20.462286 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.462262 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk"] Jul 16 11:42:20.466889 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.466860 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8bae7d/rate-test-model-8bae7d-kserve-564d74cb59-2lsxk"] Jul 16 11:42:20.886399 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.886363 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e" path="/var/lib/kubelet/pods/fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e/volumes" Jul 16 11:42:20.932459 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:20.932404 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" podUID="f05eec47-32db-46bd-af76-7a8e4d995b89" containerName="main" containerID="cri-o://c487ea42886c77ad5882316e2f583da87d724b4e6e3fdf1a242f72260eb942dd" gracePeriod=45 Jul 16 11:42:21.183463 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.183431 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:42:21.241096 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.241071 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-model-cache\") pod \"f05eec47-32db-46bd-af76-7a8e4d995b89\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " Jul 16 11:42:21.241261 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.241149 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-dshm\") pod \"f05eec47-32db-46bd-af76-7a8e4d995b89\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " Jul 16 11:42:21.241261 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.241187 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5d6v\" (UniqueName: \"kubernetes.io/projected/f05eec47-32db-46bd-af76-7a8e4d995b89-kube-api-access-f5d6v\") pod \"f05eec47-32db-46bd-af76-7a8e4d995b89\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " Jul 16 11:42:21.241261 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.241205 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f05eec47-32db-46bd-af76-7a8e4d995b89-tls-certs\") pod \"f05eec47-32db-46bd-af76-7a8e4d995b89\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " Jul 16 11:42:21.241261 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.241234 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-home\") pod \"f05eec47-32db-46bd-af76-7a8e4d995b89\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " Jul 16 11:42:21.241261 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.241260 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-tmp-dir\") pod \"f05eec47-32db-46bd-af76-7a8e4d995b89\" (UID: \"f05eec47-32db-46bd-af76-7a8e4d995b89\") " Jul 16 11:42:21.241654 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.241331 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-model-cache" (OuterVolumeSpecName: "model-cache") pod "f05eec47-32db-46bd-af76-7a8e4d995b89" (UID: "f05eec47-32db-46bd-af76-7a8e4d995b89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:42:21.241654 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.241463 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-home" (OuterVolumeSpecName: "home") pod "f05eec47-32db-46bd-af76-7a8e4d995b89" (UID: "f05eec47-32db-46bd-af76-7a8e4d995b89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:42:21.241654 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.241464 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-model-cache\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:21.241654 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.241523 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f05eec47-32db-46bd-af76-7a8e4d995b89" (UID: "f05eec47-32db-46bd-af76-7a8e4d995b89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:42:21.243401 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.243377 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-dshm" (OuterVolumeSpecName: "dshm") pod "f05eec47-32db-46bd-af76-7a8e4d995b89" (UID: "f05eec47-32db-46bd-af76-7a8e4d995b89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:42:21.243401 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.243392 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05eec47-32db-46bd-af76-7a8e4d995b89-kube-api-access-f5d6v" (OuterVolumeSpecName: "kube-api-access-f5d6v") pod "f05eec47-32db-46bd-af76-7a8e4d995b89" (UID: "f05eec47-32db-46bd-af76-7a8e4d995b89"). InnerVolumeSpecName "kube-api-access-f5d6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:42:21.243536 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.243425 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f05eec47-32db-46bd-af76-7a8e4d995b89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f05eec47-32db-46bd-af76-7a8e4d995b89" (UID: "f05eec47-32db-46bd-af76-7a8e4d995b89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:42:21.342845 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.342809 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-dshm\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:21.342845 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.342838 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5d6v\" (UniqueName: \"kubernetes.io/projected/f05eec47-32db-46bd-af76-7a8e4d995b89-kube-api-access-f5d6v\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:21.342845 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.342849 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f05eec47-32db-46bd-af76-7a8e4d995b89-tls-certs\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:21.342845 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.342858 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-home\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:21.343106 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.342867 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05eec47-32db-46bd-af76-7a8e4d995b89-tmp-dir\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:21.439037 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.438942 2580 generic.go:356] "Generic (PLEG): container finished" podID="f05eec47-32db-46bd-af76-7a8e4d995b89" containerID="c487ea42886c77ad5882316e2f583da87d724b4e6e3fdf1a242f72260eb942dd" exitCode=0 Jul 16 11:42:21.439037 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.438982 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" Jul 16 11:42:21.439037 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.439030 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" event={"ID":"f05eec47-32db-46bd-af76-7a8e4d995b89","Type":"ContainerDied","Data":"c487ea42886c77ad5882316e2f583da87d724b4e6e3fdf1a242f72260eb942dd"} Jul 16 11:42:21.439290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.439066 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd" event={"ID":"f05eec47-32db-46bd-af76-7a8e4d995b89","Type":"ContainerDied","Data":"7645e3814b6209bc73c6bddb3b7a232e2a480aa2008d5e57c3431740e660531e"} Jul 16 11:42:21.439290 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.439083 2580 scope.go:117] "RemoveContainer" containerID="c487ea42886c77ad5882316e2f583da87d724b4e6e3fdf1a242f72260eb942dd" Jul 16 11:42:21.447820 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.447804 2580 scope.go:117] "RemoveContainer" containerID="c487ea42886c77ad5882316e2f583da87d724b4e6e3fdf1a242f72260eb942dd" Jul 16 11:42:21.448115 ip-10-0-142-15 kubenswrapper[2580]: E0716 11:42:21.448098 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c487ea42886c77ad5882316e2f583da87d724b4e6e3fdf1a242f72260eb942dd\": container with ID starting with c487ea42886c77ad5882316e2f583da87d724b4e6e3fdf1a242f72260eb942dd not found: ID does not exist" containerID="c487ea42886c77ad5882316e2f583da87d724b4e6e3fdf1a242f72260eb942dd" Jul 16 11:42:21.448159 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.448121 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c487ea42886c77ad5882316e2f583da87d724b4e6e3fdf1a242f72260eb942dd"} err="failed to get container status \"c487ea42886c77ad5882316e2f583da87d724b4e6e3fdf1a242f72260eb942dd\": rpc error: code = NotFound desc = could not find container \"c487ea42886c77ad5882316e2f583da87d724b4e6e3fdf1a242f72260eb942dd\": container with ID starting with c487ea42886c77ad5882316e2f583da87d724b4e6e3fdf1a242f72260eb942dd not found: ID does not exist" Jul 16 11:42:21.473437 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.473407 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd"] Jul 16 11:42:21.481286 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.481258 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a47fa3/rate-test-model-a47fa3-kserve-7b5ddcb7ff-xzhrd"] Jul 16 11:42:21.565158 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.565134 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371679974000" Jul 16 11:42:21.596866 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.596838 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7442-140371679974000_e1b40d78-2230-4333-be66-51dccc570332/test-curl-7442-140371679974000/0.log" Jul 16 11:42:21.626383 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.626352 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7442-140371679974000"] Jul 16 11:42:21.638370 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.638340 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7442-140371679974000"] Jul 16 11:42:21.644925 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.644903 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kllv\" (UniqueName: \"kubernetes.io/projected/e1b40d78-2230-4333-be66-51dccc570332-kube-api-access-5kllv\") pod \"e1b40d78-2230-4333-be66-51dccc570332\" (UID: \"e1b40d78-2230-4333-be66-51dccc570332\") " Jul 16 11:42:21.646898 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.646875 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b40d78-2230-4333-be66-51dccc570332-kube-api-access-5kllv" (OuterVolumeSpecName: "kube-api-access-5kllv") pod "e1b40d78-2230-4333-be66-51dccc570332" (UID: "e1b40d78-2230-4333-be66-51dccc570332"). InnerVolumeSpecName "kube-api-access-5kllv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:42:21.746230 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.746203 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kllv\" (UniqueName: \"kubernetes.io/projected/e1b40d78-2230-4333-be66-51dccc570332-kube-api-access-5kllv\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:21.761081 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.761050 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7442-140371679974000"] Jul 16 11:42:21.761394 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.761381 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f05eec47-32db-46bd-af76-7a8e4d995b89" containerName="main" Jul 16 11:42:21.761436 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.761396 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f05eec47-32db-46bd-af76-7a8e4d995b89" containerName="main" Jul 16 11:42:21.761436 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.761406 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1b40d78-2230-4333-be66-51dccc570332" containerName="test-curl-7442-140371679974000" Jul 16 11:42:21.761436 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.761412 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1b40d78-2230-4333-be66-51dccc570332" containerName="test-curl-7442-140371679974000" Jul 16 11:42:21.761436 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.761423 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e" containerName="main" Jul 16 11:42:21.761436 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.761428 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e" containerName="main" Jul 16 11:42:21.761604 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.761503 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc5bdfd2-8d8a-410d-b84d-30a1dce10a1e" containerName="main" Jul 16 11:42:21.761604 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.761511 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1b40d78-2230-4333-be66-51dccc570332" containerName="test-curl-7442-140371679974000" Jul 16 11:42:21.761604 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.761517 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f05eec47-32db-46bd-af76-7a8e4d995b89" containerName="main" Jul 16 11:42:21.764323 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.764308 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371679974000" Jul 16 11:42:21.772043 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.772014 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140371679974000"] Jul 16 11:42:21.847386 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.847352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs9kt\" (UniqueName: \"kubernetes.io/projected/ecfaf284-c398-45ef-b420-6d8214a3fa71-kube-api-access-qs9kt\") pod \"test-curl-7442-140371679974000\" (UID: \"ecfaf284-c398-45ef-b420-6d8214a3fa71\") " pod="opendatahub/test-curl-7442-140371679974000" Jul 16 11:42:21.948716 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.948687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs9kt\" (UniqueName: \"kubernetes.io/projected/ecfaf284-c398-45ef-b420-6d8214a3fa71-kube-api-access-qs9kt\") pod \"test-curl-7442-140371679974000\" (UID: \"ecfaf284-c398-45ef-b420-6d8214a3fa71\") " pod="opendatahub/test-curl-7442-140371679974000" Jul 16 11:42:21.962652 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:21.962629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs9kt\" (UniqueName: \"kubernetes.io/projected/ecfaf284-c398-45ef-b420-6d8214a3fa71-kube-api-access-qs9kt\") pod \"test-curl-7442-140371679974000\" (UID: \"ecfaf284-c398-45ef-b420-6d8214a3fa71\") " pod="opendatahub/test-curl-7442-140371679974000" Jul 16 11:42:22.073642 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:22.073610 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371679974000" Jul 16 11:42:22.217653 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:22.217620 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140371679974000"] Jul 16 11:42:22.219694 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:42:22.219668 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecfaf284_c398_45ef_b420_6d8214a3fa71.slice/crio-1fd6936ccca5c26cbdb8520eaea56d7a655ca1f4251149b70bad4bd843f60503 WatchSource:0}: Error finding container 1fd6936ccca5c26cbdb8520eaea56d7a655ca1f4251149b70bad4bd843f60503: Status 404 returned error can't find the container with id 1fd6936ccca5c26cbdb8520eaea56d7a655ca1f4251149b70bad4bd843f60503 Jul 16 11:42:22.444991 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:22.444906 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140371679974000" event={"ID":"ecfaf284-c398-45ef-b420-6d8214a3fa71","Type":"ContainerStarted","Data":"1fd6936ccca5c26cbdb8520eaea56d7a655ca1f4251149b70bad4bd843f60503"} Jul 16 11:42:22.446880 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:22.446852 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="203646370bbc0763b329bce95fb7baf45b235a0ee0eb4ec9060c5f11dd6c6c19" Jul 16 11:42:22.446994 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:22.446891 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371679974000" Jul 16 11:42:22.458361 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:22.458333 2580 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7442-140371679974000" oldPodUID="e1b40d78-2230-4333-be66-51dccc570332" podUID="ecfaf284-c398-45ef-b420-6d8214a3fa71" Jul 16 11:42:22.887217 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:22.887177 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1b40d78-2230-4333-be66-51dccc570332" path="/var/lib/kubelet/pods/e1b40d78-2230-4333-be66-51dccc570332/volumes" Jul 16 11:42:22.887707 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:22.887687 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f05eec47-32db-46bd-af76-7a8e4d995b89" path="/var/lib/kubelet/pods/f05eec47-32db-46bd-af76-7a8e4d995b89/volumes" Jul 16 11:42:23.452555 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:23.452520 2580 generic.go:356] "Generic (PLEG): container finished" podID="ecfaf284-c398-45ef-b420-6d8214a3fa71" containerID="4d3d793f082bd22330d2f636952dcace44645efa96b0d68dd9b9ce1570fef3fd" exitCode=0 Jul 16 11:42:23.452953 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:23.452607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140371679974000" event={"ID":"ecfaf284-c398-45ef-b420-6d8214a3fa71","Type":"ContainerDied","Data":"4d3d793f082bd22330d2f636952dcace44645efa96b0d68dd9b9ce1570fef3fd"} Jul 16 11:42:24.590426 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:24.590401 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371679974000" Jul 16 11:42:24.613237 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:24.613215 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7442-140371679974000_ecfaf284-c398-45ef-b420-6d8214a3fa71/test-curl-7442-140371679974000/0.log" Jul 16 11:42:24.641236 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:24.641201 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7442-140371679974000"] Jul 16 11:42:24.648031 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:24.648002 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7442-140371679974000"] Jul 16 11:42:24.677087 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:24.677063 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs9kt\" (UniqueName: \"kubernetes.io/projected/ecfaf284-c398-45ef-b420-6d8214a3fa71-kube-api-access-qs9kt\") pod \"ecfaf284-c398-45ef-b420-6d8214a3fa71\" (UID: \"ecfaf284-c398-45ef-b420-6d8214a3fa71\") " Jul 16 11:42:24.679078 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:24.679053 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfaf284-c398-45ef-b420-6d8214a3fa71-kube-api-access-qs9kt" (OuterVolumeSpecName: "kube-api-access-qs9kt") pod "ecfaf284-c398-45ef-b420-6d8214a3fa71" (UID: "ecfaf284-c398-45ef-b420-6d8214a3fa71"). InnerVolumeSpecName "kube-api-access-qs9kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:42:24.777649 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:24.777608 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs9kt\" (UniqueName: \"kubernetes.io/projected/ecfaf284-c398-45ef-b420-6d8214a3fa71-kube-api-access-qs9kt\") on node \"ip-10-0-142-15.ec2.internal\" DevicePath \"\"" Jul 16 11:42:24.890723 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:24.887545 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecfaf284-c398-45ef-b420-6d8214a3fa71" path="/var/lib/kubelet/pods/ecfaf284-c398-45ef-b420-6d8214a3fa71/volumes" Jul 16 11:42:25.462409 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:25.462376 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140371679974000" Jul 16 11:42:25.462608 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:25.462417 2580 scope.go:117] "RemoveContainer" containerID="4d3d793f082bd22330d2f636952dcace44645efa96b0d68dd9b9ce1570fef3fd" Jul 16 11:42:34.785299 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:34.785267 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:42:45.885433 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:45.885339 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dzswm"] Jul 16 11:42:52.581844 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:52.581813 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58ffff6c5d-lxpk7_b5d927bf-89d0-477c-8753-5f1a218e9c29/copy-manifests/0.log" Jul 16 11:42:52.590154 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:52.590121 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58ffff6c5d-lxpk7_b5d927bf-89d0-477c-8753-5f1a218e9c29/manager/0.log" Jul 16 11:42:55.498879 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:55.498845 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-dzswm_41574ee0-18ac-4784-b08d-4d6f89cd7852/limitador/0.log" Jul 16 11:42:55.622583 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:55.622544 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x768f_68d3de97-ebf5-4eef-9dd1-85bfb37060e3/manager/0.log" Jul 16 11:42:56.362373 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:56.362345 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b4b6f7dfc-zb866_77f021cc-9bcf-41a7-8174-661b76023a27/kube-auth-proxy/0.log" Jul 16 11:42:57.817019 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:42:57.816967 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-m6mgr_da0e3b12-bceb-4cc1-aab0-d863cd4566f6/main/0.log" Jul 16 11:43:03.814439 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:03.814403 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kdtsf/must-gather-kpq4n"] Jul 16 11:43:03.814847 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:03.814805 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecfaf284-c398-45ef-b420-6d8214a3fa71" containerName="test-curl-7442-140371679974000" Jul 16 11:43:03.814847 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:03.814816 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecfaf284-c398-45ef-b420-6d8214a3fa71" containerName="test-curl-7442-140371679974000" Jul 16 11:43:03.814934 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:03.814873 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecfaf284-c398-45ef-b420-6d8214a3fa71" containerName="test-curl-7442-140371679974000" Jul 16 11:43:03.817657 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:03.817636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdtsf/must-gather-kpq4n" Jul 16 11:43:03.820267 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:03.820247 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kdtsf\"/\"openshift-service-ca.crt\"" Jul 16 11:43:03.820373 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:03.820339 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kdtsf\"/\"kube-root-ca.crt\"" Jul 16 11:43:03.820444 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:03.820371 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kdtsf\"/\"default-dockercfg-4h66r\"" Jul 16 11:43:03.828885 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:03.828860 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdtsf/must-gather-kpq4n"] Jul 16 11:43:03.936630 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:03.936592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxq2j\" (UniqueName: \"kubernetes.io/projected/0ecf16e5-8084-45fa-b8f6-3b6dd52e602b-kube-api-access-cxq2j\") pod \"must-gather-kpq4n\" (UID: \"0ecf16e5-8084-45fa-b8f6-3b6dd52e602b\") " pod="openshift-must-gather-kdtsf/must-gather-kpq4n" Jul 16 11:43:03.936802 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:03.936702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0ecf16e5-8084-45fa-b8f6-3b6dd52e602b-must-gather-output\") pod \"must-gather-kpq4n\" (UID: \"0ecf16e5-8084-45fa-b8f6-3b6dd52e602b\") " pod="openshift-must-gather-kdtsf/must-gather-kpq4n" Jul 16 11:43:04.037333 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:04.037279 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0ecf16e5-8084-45fa-b8f6-3b6dd52e602b-must-gather-output\") pod \"must-gather-kpq4n\" (UID: \"0ecf16e5-8084-45fa-b8f6-3b6dd52e602b\") " pod="openshift-must-gather-kdtsf/must-gather-kpq4n" Jul 16 11:43:04.037333 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:04.037332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxq2j\" (UniqueName: \"kubernetes.io/projected/0ecf16e5-8084-45fa-b8f6-3b6dd52e602b-kube-api-access-cxq2j\") pod \"must-gather-kpq4n\" (UID: \"0ecf16e5-8084-45fa-b8f6-3b6dd52e602b\") " pod="openshift-must-gather-kdtsf/must-gather-kpq4n" Jul 16 11:43:04.037658 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:04.037637 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0ecf16e5-8084-45fa-b8f6-3b6dd52e602b-must-gather-output\") pod \"must-gather-kpq4n\" (UID: \"0ecf16e5-8084-45fa-b8f6-3b6dd52e602b\") " pod="openshift-must-gather-kdtsf/must-gather-kpq4n" Jul 16 11:43:04.046068 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:04.046045 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxq2j\" (UniqueName: \"kubernetes.io/projected/0ecf16e5-8084-45fa-b8f6-3b6dd52e602b-kube-api-access-cxq2j\") pod \"must-gather-kpq4n\" (UID: \"0ecf16e5-8084-45fa-b8f6-3b6dd52e602b\") " pod="openshift-must-gather-kdtsf/must-gather-kpq4n" Jul 16 11:43:04.128299 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:04.128206 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdtsf/must-gather-kpq4n" Jul 16 11:43:04.260670 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:04.260642 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdtsf/must-gather-kpq4n"] Jul 16 11:43:04.263099 ip-10-0-142-15 kubenswrapper[2580]: W0716 11:43:04.263068 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ecf16e5_8084_45fa_b8f6_3b6dd52e602b.slice/crio-6bd11975dfed87a6ea3b6d5a3f75b53e83eedf38e54096fa0657deb83b91a453 WatchSource:0}: Error finding container 6bd11975dfed87a6ea3b6d5a3f75b53e83eedf38e54096fa0657deb83b91a453: Status 404 returned error can't find the container with id 6bd11975dfed87a6ea3b6d5a3f75b53e83eedf38e54096fa0657deb83b91a453 Jul 16 11:43:04.659387 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:04.659351 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdtsf/must-gather-kpq4n" event={"ID":"0ecf16e5-8084-45fa-b8f6-3b6dd52e602b","Type":"ContainerStarted","Data":"6bd11975dfed87a6ea3b6d5a3f75b53e83eedf38e54096fa0657deb83b91a453"} Jul 16 11:43:05.669181 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:05.668435 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdtsf/must-gather-kpq4n" event={"ID":"0ecf16e5-8084-45fa-b8f6-3b6dd52e602b","Type":"ContainerStarted","Data":"9b540c2cd1ba376a483a4d569853f8b2283afea663d8a8c1887caa367698f6b5"} Jul 16 11:43:05.669181 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:05.668486 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdtsf/must-gather-kpq4n" event={"ID":"0ecf16e5-8084-45fa-b8f6-3b6dd52e602b","Type":"ContainerStarted","Data":"4d041e63f5abc09e5917e6e5cb4e7752e7e0cb5fe67bd7f9032f1849bfa5b76a"} Jul 16 11:43:05.686365 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:05.686312 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kdtsf/must-gather-kpq4n" podStartSLOduration=1.8580612570000001 podStartE2EDuration="2.68629562s" totalImagesPullingTime="828.234363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:43:05.684756093 +0000 UTC m=+2357.518104777" watchObservedRunningTime="2026-07-16 11:43:05.68629562 +0000 UTC m=+2357.519644313" Jul 16 11:43:06.707484 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:06.707455 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fxlr6_25a48584-b83e-4cff-a68f-18c383e81fc7/global-pull-secret-syncer/0.log" Jul 16 11:43:06.892301 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:06.892266 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mn5mk_2e5ba1b3-7981-43d4-b14a-9ace2f2e1128/konnectivity-agent/0.log" Jul 16 11:43:07.002891 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:07.002863 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-15.ec2.internal_b2e3677082925fab7d36c1bc7a29637d/haproxy/0.log" Jul 16 11:43:12.954250 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:12.954208 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-dzswm_41574ee0-18ac-4784-b08d-4d6f89cd7852/limitador/0.log" Jul 16 11:43:13.043428 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:13.043388 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x768f_68d3de97-ebf5-4eef-9dd1-85bfb37060e3/manager/0.log" Jul 16 11:43:14.548676 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:14.548596 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f0f0c71-ac01-4c43-8464-9e8c4bd2056a/alertmanager/0.log" Jul 16 11:43:14.594034 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:14.593995 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f0f0c71-ac01-4c43-8464-9e8c4bd2056a/config-reloader/0.log" Jul 16 11:43:14.620286 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:14.620213 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f0f0c71-ac01-4c43-8464-9e8c4bd2056a/kube-rbac-proxy-web/0.log" Jul 16 11:43:14.649414 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:14.649383 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f0f0c71-ac01-4c43-8464-9e8c4bd2056a/kube-rbac-proxy/0.log" Jul 16 11:43:14.675241 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:14.675213 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f0f0c71-ac01-4c43-8464-9e8c4bd2056a/kube-rbac-proxy-metric/0.log" Jul 16 11:43:14.703155 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:14.703128 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f0f0c71-ac01-4c43-8464-9e8c4bd2056a/prom-label-proxy/0.log" Jul 16 11:43:14.731629 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:14.731603 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f0f0c71-ac01-4c43-8464-9e8c4bd2056a/init-config-reloader/0.log" Jul 16 11:43:14.806484 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:14.806454 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-5sqjf_bb8e1f89-1dc4-49b7-9b40-980230ee378f/kube-state-metrics/0.log" Jul 16 11:43:14.828804 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:14.828774 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-5sqjf_bb8e1f89-1dc4-49b7-9b40-980230ee378f/kube-rbac-proxy-main/0.log" Jul 16 11:43:14.856109 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:14.856087 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-5sqjf_bb8e1f89-1dc4-49b7-9b40-980230ee378f/kube-rbac-proxy-self/0.log" Jul 16 11:43:14.885830 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:14.885744 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-76c4dbdb5b-jhthw_347220ac-836a-4ec0-8036-d639a8a494fd/metrics-server/0.log" Jul 16 11:43:14.958226 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:14.958189 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4j8r2_184d37e6-e16f-4f5f-9c83-c423c8c53d4a/node-exporter/0.log" Jul 16 11:43:14.984849 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:14.984777 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4j8r2_184d37e6-e16f-4f5f-9c83-c423c8c53d4a/kube-rbac-proxy/0.log" Jul 16 11:43:15.011986 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.011886 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4j8r2_184d37e6-e16f-4f5f-9c83-c423c8c53d4a/init-textfile/0.log" Jul 16 11:43:15.211957 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.211870 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-qx9k5_d8fa8508-ba77-46f9-b371-c935f10d0740/kube-rbac-proxy-main/0.log" Jul 16 11:43:15.238560 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.238527 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-qx9k5_d8fa8508-ba77-46f9-b371-c935f10d0740/kube-rbac-proxy-self/0.log" Jul 16 11:43:15.263662 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.263627 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-qx9k5_d8fa8508-ba77-46f9-b371-c935f10d0740/openshift-state-metrics/0.log" Jul 16 11:43:15.312348 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.312278 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_31bfd179-2649-4908-ab65-6856930379bc/prometheus/0.log" Jul 16 11:43:15.337421 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.337374 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_31bfd179-2649-4908-ab65-6856930379bc/config-reloader/0.log" Jul 16 11:43:15.363965 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.363918 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_31bfd179-2649-4908-ab65-6856930379bc/thanos-sidecar/0.log" Jul 16 11:43:15.389053 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.389022 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_31bfd179-2649-4908-ab65-6856930379bc/kube-rbac-proxy-web/0.log" Jul 16 11:43:15.416174 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.416149 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_31bfd179-2649-4908-ab65-6856930379bc/kube-rbac-proxy/0.log" Jul 16 11:43:15.441826 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.441800 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_31bfd179-2649-4908-ab65-6856930379bc/kube-rbac-proxy-thanos/0.log" Jul 16 11:43:15.467611 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.467521 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_31bfd179-2649-4908-ab65-6856930379bc/init-config-reloader/0.log" Jul 16 11:43:15.550241 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.550209 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-jx876_90ade4f6-eb51-4a6e-b54b-f30b10701c81/prometheus-operator-admission-webhook/0.log" Jul 16 11:43:15.586525 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.586489 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74cf4d787c-qd7g7_b0a7a63e-e016-4f06-a5aa-8104fed5a765/telemeter-client/0.log" Jul 16 11:43:15.612116 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.612087 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74cf4d787c-qd7g7_b0a7a63e-e016-4f06-a5aa-8104fed5a765/reload/0.log" Jul 16 11:43:15.639418 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:15.639385 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74cf4d787c-qd7g7_b0a7a63e-e016-4f06-a5aa-8104fed5a765/kube-rbac-proxy/0.log" Jul 16 11:43:16.360000 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.359961 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm"] Jul 16 11:43:16.363674 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.363645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.373595 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.373545 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm"] Jul 16 11:43:16.470885 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.470851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppmrr\" (UniqueName: \"kubernetes.io/projected/6bebabb3-7144-4987-acdb-cff02c45efeb-kube-api-access-ppmrr\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.471063 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.470899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bebabb3-7144-4987-acdb-cff02c45efeb-sys\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.471063 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.471017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6bebabb3-7144-4987-acdb-cff02c45efeb-proc\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.471063 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.471056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6bebabb3-7144-4987-acdb-cff02c45efeb-podres\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.471219 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.471111 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bebabb3-7144-4987-acdb-cff02c45efeb-lib-modules\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.571920 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.571886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bebabb3-7144-4987-acdb-cff02c45efeb-lib-modules\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.572375 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.571945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppmrr\" (UniqueName: \"kubernetes.io/projected/6bebabb3-7144-4987-acdb-cff02c45efeb-kube-api-access-ppmrr\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.572375 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.571981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bebabb3-7144-4987-acdb-cff02c45efeb-sys\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.572375 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.572070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6bebabb3-7144-4987-acdb-cff02c45efeb-proc\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.572375 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.572087 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bebabb3-7144-4987-acdb-cff02c45efeb-lib-modules\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.572375 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.572115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6bebabb3-7144-4987-acdb-cff02c45efeb-podres\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.572375 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.572160 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bebabb3-7144-4987-acdb-cff02c45efeb-sys\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.572375 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.572211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6bebabb3-7144-4987-acdb-cff02c45efeb-proc\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.572375 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.572265 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6bebabb3-7144-4987-acdb-cff02c45efeb-podres\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.582513 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.582477 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppmrr\" (UniqueName: \"kubernetes.io/projected/6bebabb3-7144-4987-acdb-cff02c45efeb-kube-api-access-ppmrr\") pod \"perf-node-gather-daemonset-dk8dm\" (UID: \"6bebabb3-7144-4987-acdb-cff02c45efeb\") " pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.678063 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.677965 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:16.871094 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:16.869376 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm"] Jul 16 11:43:17.753992 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:17.753960 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" event={"ID":"6bebabb3-7144-4987-acdb-cff02c45efeb","Type":"ContainerStarted","Data":"5da686a0bf45788cfbb8147b6b632731dc8ddb087a316d5d9759530d328bfea6"} Jul 16 11:43:17.754389 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:17.754012 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" event={"ID":"6bebabb3-7144-4987-acdb-cff02c45efeb","Type":"ContainerStarted","Data":"8540c6215ae29fcfc95a4178f8790117bcda3ce5548091b7d076a007b5816f9c"} Jul 16 11:43:17.754389 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:17.754060 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:17.774225 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:17.774151 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" podStartSLOduration=1.774131742 podStartE2EDuration="1.774131742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:43:17.773906569 +0000 UTC m=+2369.607255268" watchObservedRunningTime="2026-07-16 11:43:17.774131742 +0000 UTC m=+2369.607480425" Jul 16 11:43:18.361828 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:18.361803 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84cf84f4b5-gbjzw_285179f9-23e7-470d-a659-49b1c5bc5a74/console/0.log" Jul 16 11:43:20.187304 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:20.187270 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ddhln_34c50701-dc77-4154-895c-cf88dfd93f5e/dns/0.log" Jul 16 11:43:20.214127 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:20.214078 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ddhln_34c50701-dc77-4154-895c-cf88dfd93f5e/kube-rbac-proxy/0.log" Jul 16 11:43:20.267177 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:20.267151 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ss9lt_daaff1e0-e805-446f-8a08-4baf8c77191d/dns-node-resolver/0.log" Jul 16 11:43:20.926420 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:20.926384 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r76fs_870684c4-b8b2-48d5-8c2f-97c635fad4df/node-ca/0.log" Jul 16 11:43:22.147592 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:22.147546 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b4b6f7dfc-zb866_77f021cc-9bcf-41a7-8174-661b76023a27/kube-auth-proxy/0.log" Jul 16 11:43:23.057326 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:23.057294 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rb76j_d6c40a24-95a6-4c49-b0b2-9140c8e05b4b/serve-healthcheck-canary/0.log" Jul 16 11:43:23.721327 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:23.721296 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nxfs5_00196644-3e96-4e0a-9002-ac8d51e53cc1/kube-rbac-proxy/0.log" Jul 16 11:43:23.745462 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:23.745435 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nxfs5_00196644-3e96-4e0a-9002-ac8d51e53cc1/exporter/0.log" Jul 16 11:43:23.770060 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:23.770034 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nxfs5_00196644-3e96-4e0a-9002-ac8d51e53cc1/extractor/0.log" Jul 16 11:43:23.773371 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:23.773348 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kdtsf/perf-node-gather-daemonset-dk8dm" Jul 16 11:43:26.895157 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:26.895085 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58ffff6c5d-lxpk7_b5d927bf-89d0-477c-8753-5f1a218e9c29/manager/0.log" Jul 16 11:43:26.920878 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:26.920849 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58ffff6c5d-lxpk7_b5d927bf-89d0-477c-8753-5f1a218e9c29/copy-manifests/0.log" Jul 16 11:43:27.053434 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:27.053405 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b4b6f7dfc-zb866_77f021cc-9bcf-41a7-8174-661b76023a27/kube-auth-proxy/0.log" Jul 16 11:43:28.352856 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:28.352741 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5dbfbcbfc5-zkb48_b276feaa-ccf3-4f4c-8e28-7523b65e19b1/manager/0.log" Jul 16 11:43:28.388296 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:28.388267 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-9m7wv_3826c964-b173-4bdb-a952-604a9631f721/openshift-lws-operator/0.log" Jul 16 11:43:28.667728 ip-10-0-142-15 kubenswrapper[2580]: I0716 11:43:28.667645 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b4b6f7dfc-zb866_77f021cc-9bcf-41a7-8174-661b76023a27/kube-auth-proxy/0.log"