Jul 10 02:22:49.454926 ip-10-0-128-69 systemd[1]: Starting Kubernetes Kubelet... Jul 10 02:22:49.877148 ip-10-0-128-69 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 02:22:49.877148 ip-10-0-128-69 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 02:22:49.877148 ip-10-0-128-69 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 02:22:49.877148 ip-10-0-128-69 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 02:22:49.877148 ip-10-0-128-69 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 02:22:49.877148 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.869652 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 02:22:49.877148 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875150 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:22:49.877148 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875164 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:22:49.877148 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875169 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:22:49.877148 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875173 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:22:49.877148 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875176 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:22:49.877148 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875179 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:22:49.877148 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875182 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:22:49.877148 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875185 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875188 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875190 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875193 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875196 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875198 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875201 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875205 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875209 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875212 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875229 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875232 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875235 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875238 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875241 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875244 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875246 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875249 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875253 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:22:49.887208 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875256 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875260 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875263 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875266 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875269 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875271 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875274 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875277 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875279 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875282 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875285 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875287 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875290 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875293 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875295 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875298 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875300 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875303 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875306 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875309 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:22:49.891098 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875312 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875315 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875317 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875320 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875322 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875325 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875328 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875330 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875333 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875336 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875338 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875341 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875344 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875346 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875349 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875351 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875354 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875356 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875360 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875362 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875365 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:22:49.906103 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875367 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875370 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875389 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875393 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875396 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875399 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875402 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875405 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875408 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875410 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875414 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875417 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875420 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875422 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875425 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875428 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875430 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875432 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875435 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875437 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:22:49.907858 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875440 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875442 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875445 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875448 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875451 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875454 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875456 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875459 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875461 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875464 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875466 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.875470 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876144 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876152 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876157 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876160 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876163 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876166 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876169 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:22:49.927578 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876171 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876174 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876177 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876180 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876183 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876186 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876188 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876191 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876194 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876196 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876199 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876201 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876204 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876207 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876210 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876212 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876215 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876217 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876220 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876222 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:22:49.930790 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876226 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876228 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876231 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876233 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876236 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876238 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876241 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876243 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876245 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876248 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876250 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876253 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876256 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876258 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876262 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876264 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876267 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876269 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876272 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876275 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:22:49.940753 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876278 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876280 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876283 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876286 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876288 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876291 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876294 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876297 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876300 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876304 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876308 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876311 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876313 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876317 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876321 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876324 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876326 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876329 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876332 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:22:49.943910 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876334 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876337 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876339 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876342 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876345 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876348 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876351 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876353 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876356 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876358 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876361 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876363 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876366 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876368 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876371 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876374 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876377 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876379 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876382 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876384 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:22:49.955361 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876387 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:22:49.955954 ip-10-0-128-69 systemd[1]: Started Kubernetes Kubelet. Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876390 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876395 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876398 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876401 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876404 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876407 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876410 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876412 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876414 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876417 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876420 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876422 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876804 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876809 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876812 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876815 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876818 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876823 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:22:49.961241 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876827 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876830 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876833 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876836 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876843 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876846 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876849 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876852 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876856 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876859 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876862 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876865 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876867 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876871 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876873 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876876 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876878 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876881 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876884 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876886 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:22:49.981894 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876889 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876891 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876894 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876897 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876900 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876902 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876905 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876907 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876910 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876913 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876915 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876919 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876921 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876924 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876927 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876929 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876934 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876937 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876939 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876942 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:22:49.985995 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876946 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876949 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876951 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876954 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876957 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876959 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876962 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876964 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876967 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876969 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876972 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876974 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876977 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876980 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876982 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876985 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876987 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876990 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876992 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876995 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:22:50.183764 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.876998 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877001 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877004 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877006 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877011 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877020 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877024 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877029 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877033 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877036 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877039 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877042 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877047 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877050 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877053 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877056 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877058 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877061 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877064 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:22:50.184963 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877067 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877070 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877072 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877076 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877078 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877081 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877084 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877086 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877089 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877091 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877094 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877097 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877099 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877101 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877178 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877188 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877197 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877204 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877212 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877217 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877223 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 02:22:50.187351 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877230 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877233 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877236 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877239 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877245 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877248 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877251 2575 flags.go:64] FLAG: --cgroup-root="" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877254 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877257 2575 flags.go:64] FLAG: --client-ca-file="" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877260 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877263 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877267 2575 flags.go:64] FLAG: --cluster-domain="" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877270 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877273 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877276 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877280 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877284 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877286 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877289 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877293 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877296 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877299 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877302 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877305 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 02:22:50.188038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877309 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877316 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877319 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877322 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877325 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877328 2575 flags.go:64] FLAG: --enable-server="true" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877332 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877336 2575 flags.go:64] FLAG: --event-burst="100" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877341 2575 flags.go:64] FLAG: --event-qps="50" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877344 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877348 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877351 2575 flags.go:64] FLAG: --eviction-hard="" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877355 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877358 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877361 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877364 2575 flags.go:64] FLAG: --eviction-soft="" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877367 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877370 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877373 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877376 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877379 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877382 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877385 2575 flags.go:64] FLAG: --feature-gates="" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877388 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877391 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 02:22:50.188895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877394 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877397 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877400 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877403 2575 flags.go:64] FLAG: --help="false" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877406 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877410 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877412 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877416 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877419 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877422 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877425 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877428 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877431 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877434 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877437 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877442 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877445 2575 flags.go:64] FLAG: --kube-reserved="" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877448 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877451 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877456 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877458 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877461 2575 flags.go:64] FLAG: --lock-file="" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877464 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877467 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 02:22:50.189684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877470 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877475 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877478 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877481 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877484 2575 flags.go:64] FLAG: --logging-format="text" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877487 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877490 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877493 2575 flags.go:64] FLAG: --manifest-url="" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877496 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877500 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877503 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877507 2575 flags.go:64] FLAG: --max-pods="110" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877510 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877512 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877515 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877518 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877521 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877524 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877527 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877534 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877538 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877541 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877544 2575 flags.go:64] FLAG: --pod-cidr="" Jul 10 02:22:50.190707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877548 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877553 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877556 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877559 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877563 2575 flags.go:64] FLAG: --port="10250" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877567 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877570 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03fa0dc405fe80f70" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877573 2575 flags.go:64] FLAG: --qos-reserved="" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877576 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877579 2575 flags.go:64] FLAG: --register-node="true" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877582 2575 flags.go:64] FLAG: --register-with-taints="" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877586 2575 flags.go:64] FLAG: --registry-burst="10" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877589 2575 flags.go:64] FLAG: --registry-qps="5" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877591 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877594 2575 flags.go:64] FLAG: --reserved-memory="" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877612 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877615 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877619 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877622 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877624 2575 flags.go:64] FLAG: --runonce="false" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877627 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877631 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877634 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877637 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877639 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877642 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 02:22:50.191415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877645 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877648 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877652 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877655 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877658 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877660 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877663 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877669 2575 flags.go:64] FLAG: --system-cgroups="" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877672 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877677 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877681 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877684 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877688 2575 flags.go:64] FLAG: --tls-min-version="" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877691 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877694 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877697 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877700 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877703 2575 flags.go:64] FLAG: --v="2" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877707 2575 flags.go:64] FLAG: --version="false" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877711 2575 flags.go:64] FLAG: --vmodule="" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877715 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.877718 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877834 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877837 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877840 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:22:50.192517 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877843 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877846 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877848 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877852 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877854 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877857 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877860 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877863 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877866 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877868 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877871 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877874 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877876 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877879 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877883 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877889 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877892 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877897 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877899 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877902 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:22:50.193283 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877905 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877907 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877910 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877913 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877916 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877918 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877921 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877924 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877926 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877929 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877932 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877934 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877937 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877940 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877943 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877945 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877948 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877950 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877953 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877956 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:22:50.193881 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877958 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877961 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877963 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877966 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877969 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877971 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877975 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877978 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877981 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877985 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877987 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877990 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877993 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877995 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.877998 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878001 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878003 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878006 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878008 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878011 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:22:50.194725 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878014 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878016 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878019 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878021 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878024 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878027 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878031 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878034 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878038 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878041 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878045 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878048 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878051 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878053 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878057 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878061 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878064 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878067 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:22:50.195296 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878071 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878074 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878077 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878082 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878084 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878087 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878090 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878092 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878095 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878098 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878100 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878103 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878105 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878109 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878111 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878114 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878116 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:22:50.195895 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.878119 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.878684 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.886356 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.886370 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.886386 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.886393 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886462 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886467 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886471 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886474 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886477 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886480 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886483 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886486 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886489 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:22:50.196475 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886492 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886498 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886501 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886505 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886508 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886511 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886514 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886516 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886519 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886522 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886524 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886527 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886529 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886532 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886534 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886537 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886540 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886542 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886545 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886548 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:22:50.197113 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886551 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886554 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886557 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886561 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886565 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886567 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886570 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886573 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886575 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886578 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886581 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886583 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886586 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886589 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886592 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886613 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886616 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886619 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886622 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:22:50.412328 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886625 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886627 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886630 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886633 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886635 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886640 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886643 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886646 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886649 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886652 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886655 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886658 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886661 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886663 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886667 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886669 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886672 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886675 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886677 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:22:50.413132 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886680 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886682 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886685 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886687 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886690 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886693 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886695 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886700 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886702 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886706 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886709 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886711 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886714 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886717 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886720 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886722 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886725 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886727 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886730 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886733 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:22:50.413937 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886735 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886738 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886740 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886743 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886747 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886750 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886753 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886756 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886759 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886762 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886765 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886767 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.886772 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886880 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886884 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:22:50.414469 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886887 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886890 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886893 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886896 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886898 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886902 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886906 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886909 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886911 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886914 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886916 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886919 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886922 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886924 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886927 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886929 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886932 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886934 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886937 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886939 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:22:50.632923 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886942 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886946 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886949 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886952 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886956 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886959 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886962 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886965 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886967 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886970 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886973 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886976 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886978 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886981 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886984 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886986 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886988 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886994 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.886998 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887000 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:22:50.634184 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887003 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887005 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887008 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887011 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887014 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887016 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887019 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887021 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887024 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887026 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887029 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887032 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887034 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887037 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887040 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887042 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887045 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887047 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887050 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887053 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:22:50.634788 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887056 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887058 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887061 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887063 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887066 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887068 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887071 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887074 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887076 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887080 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887084 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887086 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887089 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887091 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887094 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887097 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887099 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887102 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887104 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887107 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:22:50.635405 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887109 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887112 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887115 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887117 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887120 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887122 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887125 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887127 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887131 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887135 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887138 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887141 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887144 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887146 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887150 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887153 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:22:50.635970 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:49.887156 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:22:50.636472 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.887161 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 02:22:50.636472 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.887256 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 02:22:50.636472 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.889388 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 02:22:50.636472 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.891051 2575 server.go:1019] "Starting client certificate rotation" Jul 10 02:22:50.636472 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.891154 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 02:22:50.636472 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.891192 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 02:22:50.636472 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.894247 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 02:22:50.636472 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.901628 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 02:22:50.636472 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.916994 2575 log.go:25] "Validated CRI v1 runtime API" Jul 10 02:22:50.636472 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.919032 2575 log.go:25] "Validated CRI v1 image API" Jul 10 02:22:50.636472 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.920466 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 02:22:50.636472 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.923011 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 02:22:50.636864 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.926284 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ee9ffcf2-f209-40d8-956b-66e70dd9de01:/dev/nvme0n1p4 f76b724c-2be7-49e3-b9bd-339badac5226:/dev/nvme0n1p3] Jul 10 02:22:50.636864 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.926301 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.931968 2575 manager.go:216] Machine: {Timestamp:2026-07-10 02:22:49.930747217 +0000 UTC m=+0.354845202 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103725 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b97ae99196ed1e091ebc47066b134 SystemUUID:ec2b97ae-9919-6ed1-e091-ebc47066b134 BootID:4cb37735-a204-4d44-b643-66552f7a28c2 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:dd:62:80:a8:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:dd:62:80:a8:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:3d:fd:d5:2f:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.932074 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.932188 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.935195 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.935218 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-69.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.936469 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.936480 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.936493 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.937194 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.938139 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.941035 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.941153 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.941191 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.941211 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.941233 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.941246 2575 kubelet.go:398] "Adding apiserver pod source" Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.941258 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 02:22:50.637184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.941676 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.941692 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.944972 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.946707 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.946774 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.947479 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.947493 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.947507 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.947518 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.947532 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.947537 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.947549 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.947554 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.947560 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.947571 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.947589 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.947594 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.948555 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.948567 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:49.954011 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-69.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:49.954012 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.954046 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-69.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.954138 2575 server.go:1268] "Started kubelet" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.954747 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.955372 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.955441 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.962019 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:49.963753 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.966081 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.966103 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:49.966865 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-69.ec2.internal\" not found" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.967078 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.967111 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.967188 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.967427 2575 factory.go:55] Registering systemd factory Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.967451 2575 factory.go:223] Registration of the systemd container factory successfully Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.967467 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.967475 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:49.967375 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-69.ec2.internal.18c0cc611fae1e7d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-69.ec2.internal,UID:ip-10-0-128-69.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-69.ec2.internal,},FirstTimestamp:2026-07-10 02:22:49.954115197 +0000 UTC m=+0.378213182,LastTimestamp:2026-07-10 02:22:49.954115197 +0000 UTC m=+0.378213182,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-69.ec2.internal,}" Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.970435 2575 factory.go:153] Registering CRI-O factory Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.970452 2575 factory.go:223] Registration of the crio container factory successfully Jul 10 02:22:50.637716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.970486 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.970504 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.970543 2575 factory.go:103] Registering Raw factory Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.970555 2575 manager.go:1196] Started watching for new ooms in manager Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:49.972578 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-69.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:49.972880 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.974220 2575 manager.go:349] Starting recovery of all containers Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.989505 2575 manager.go:354] Recovery completed Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.997893 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.999750 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.999778 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:49.999788 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.000371 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.000381 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.000383 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.000397 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.001100 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.001124 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.001133 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.001887 2575 policy_none.go:49] "None policy: Start" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.001901 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.001912 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.002115 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-69.ec2.internal.18c0cc612266adbb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-69.ec2.internal,UID:ip-10-0-128-69.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-69.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-69.ec2.internal,},FirstTimestamp:2026-07-10 02:22:49.999764923 +0000 UTC m=+0.423862908,LastTimestamp:2026-07-10 02:22:49.999764923 +0000 UTC m=+0.423862908,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-69.ec2.internal,}" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.003314 2575 policy_none.go:47] "Start" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.010782 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-69.ec2.internal.18c0cc612266f3a1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-69.ec2.internal,UID:ip-10-0-128-69.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-69.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-69.ec2.internal,},FirstTimestamp:2026-07-10 02:22:49.999782817 +0000 UTC m=+0.423880801,LastTimestamp:2026-07-10 02:22:49.999782817 +0000 UTC m=+0.423880801,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-69.ec2.internal,}" Jul 10 02:22:50.638741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.012533 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2msw5" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.013399 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-69.ec2.internal.18c0cc6122671916 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-69.ec2.internal,UID:ip-10-0-128-69.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-69.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-69.ec2.internal,},FirstTimestamp:2026-07-10 02:22:49.999792406 +0000 UTC m=+0.423890390,LastTimestamp:2026-07-10 02:22:49.999792406 +0000 UTC m=+0.423890390,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-69.ec2.internal,}" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.022471 2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-69.ec2.internal.18c0cc612266adbb\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-69.ec2.internal.18c0cc612266adbb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-69.ec2.internal,UID:ip-10-0-128-69.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-69.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-69.ec2.internal,},FirstTimestamp:2026-07-10 02:22:49.999764923 +0000 UTC m=+0.423862908,LastTimestamp:2026-07-10 02:22:50.001110432 +0000 UTC m=+0.425208416,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-69.ec2.internal,}" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.025075 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2msw5" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.038159 2575 manager.go:337] "Starting Device Plugin manager" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.038191 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.038204 2575 server.go:87] "Starting device plugin registration server" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.038509 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.038522 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.038659 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.038734 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.038742 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.039398 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.039435 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-69.ec2.internal\" not found" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.096996 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.098196 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.098241 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.098266 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.098307 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.100880 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.139059 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.139939 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.139974 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.139989 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.140025 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.147366 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.147388 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-69.ec2.internal\": node \"ip-10-0-128-69.ec2.internal\" not found" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.160875 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-69.ec2.internal\" not found" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.198520 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-69.ec2.internal"] Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.198579 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.199652 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:22:50.639368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.199680 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.199699 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.200966 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.201056 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.201099 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.205083 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.205120 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.207542 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.207559 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.207565 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.207574 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.209169 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.209190 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.209900 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.209923 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.209935 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.243204 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-69.ec2.internal\" not found" node="ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.247733 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-69.ec2.internal\" not found" node="ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.261962 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-69.ec2.internal\" not found" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.269343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174a0477fbdfffd07715b9db16c916ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal\" (UID: \"174a0477fbdfffd07715b9db16c916ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.269370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/042669854574a08b037a614d9f1b2909-config\") pod \"kube-apiserver-proxy-ip-10-0-128-69.ec2.internal\" (UID: \"042669854574a08b037a614d9f1b2909\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.269386 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/174a0477fbdfffd07715b9db16c916ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal\" (UID: \"174a0477fbdfffd07715b9db16c916ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.362161 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-69.ec2.internal\" not found" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.370517 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/174a0477fbdfffd07715b9db16c916ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal\" (UID: \"174a0477fbdfffd07715b9db16c916ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.370542 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174a0477fbdfffd07715b9db16c916ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal\" (UID: \"174a0477fbdfffd07715b9db16c916ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.370580 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/042669854574a08b037a614d9f1b2909-config\") pod \"kube-apiserver-proxy-ip-10-0-128-69.ec2.internal\" (UID: \"042669854574a08b037a614d9f1b2909\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.370644 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/174a0477fbdfffd07715b9db16c916ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal\" (UID: \"174a0477fbdfffd07715b9db16c916ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640804 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.370643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174a0477fbdfffd07715b9db16c916ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal\" (UID: \"174a0477fbdfffd07715b9db16c916ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640804 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.370646 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/042669854574a08b037a614d9f1b2909-config\") pod \"kube-apiserver-proxy-ip-10-0-128-69.ec2.internal\" (UID: \"042669854574a08b037a614d9f1b2909\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640804 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.462911 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-69.ec2.internal\" not found" Jul 10 02:22:50.640804 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.545454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640804 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:50.550302 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.ec2.internal" Jul 10 02:22:50.640804 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.563935 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-69.ec2.internal\" not found" Jul 10 02:22:50.664565 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:50.664514 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-69.ec2.internal\" not found" Jul 10 02:22:51.789751 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:51.789710 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-69.ec2.internal\" not found" Jul 10 02:22:51.818848 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.818823 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:22:51.847018 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.846992 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:22:51.891886 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.891720 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal" Jul 10 02:22:51.904496 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.904474 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 02:22:51.906771 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.906752 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.ec2.internal" Jul 10 02:22:51.914611 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.914591 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 02:22:51.914740 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.914721 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 02:22:51.914803 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.914744 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 02:22:51.914803 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.914744 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 02:22:51.914803 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:51.914779 2575 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://af3c4259be5984dc3b355f89efdda0c5-4eac27ba12f008f1.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.128.69:34814->100.50.34.89:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.ec2.internal" Jul 10 02:22:51.966927 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.966877 2575 apiserver.go:52] "Watching apiserver" Jul 10 02:22:51.985970 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.985951 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 02:22:51.986998 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.986979 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-sd54c","openshift-ovn-kubernetes/ovnkube-node-nlm7j","kube-system/global-pull-secret-syncer-lgw26","kube-system/konnectivity-agent-mnprz","kube-system/kube-apiserver-proxy-ip-10-0-128-69.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal","openshift-multus/multus-blpm7","openshift-multus/network-metrics-daemon-r4t9z","openshift-network-diagnostics/network-check-target-j28gk","openshift-network-operator/iptables-alerter-mrv4l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt","openshift-cluster-node-tuning-operator/tuned-gxxlp","openshift-dns/node-resolver-d47h5","openshift-image-registry/node-ca-hv4z2"] Jul 10 02:22:51.988789 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.988769 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:51.990257 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.990222 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.990912 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.990894 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 02:22:51.991617 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.991603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:51.991680 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:51.991664 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:22:51.992625 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.992610 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 02:22:51.992625 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.992618 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 02:22:51.992926 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.992904 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mnprz" Jul 10 02:22:51.993308 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.993134 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 02:22:51.993308 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.993285 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 02:22:51.993462 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.993319 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 02:22:51.993462 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.993397 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 02:22:51.994463 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.994448 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-blpm7" Jul 10 02:22:51.994589 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.994568 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 02:22:51.994739 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.994624 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-984g8\"" Jul 10 02:22:51.994739 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.994634 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 02:22:51.994953 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.994939 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 02:22:51.995526 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.995512 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cw7z5\"" Jul 10 02:22:51.995591 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.995513 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 02:22:51.995878 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.995865 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 02:22:51.996123 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.996108 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t2lcm\"" Jul 10 02:22:51.996322 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.996306 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:22:51.996392 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:51.996360 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:22:51.997566 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.997548 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:22:51.997652 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:51.997599 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:22:51.997947 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.997931 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 02:22:51.997947 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.997945 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 02:22:51.998103 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.997949 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 02:22:51.998103 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.997987 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2lkjl\"" Jul 10 02:22:51.998274 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998241 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82138b66-c754-4c2d-a828-a104a92a9968-dbus\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:51.998329 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998287 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-run-ovn-kubernetes\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.998329 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61241733-b531-4151-a869-cc89000e2e99-ovnkube-config\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.998433 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998335 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61241733-b531-4151-a869-cc89000e2e99-ovnkube-script-lib\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.998433 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d74a1be9-254d-4ec7-a66a-e521e4d19dca-cni-binary-copy\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:51.998433 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-var-lib-openvswitch\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.998433 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998419 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-log-socket\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.998632 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998464 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:51.998632 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998489 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltx8m\" (UniqueName: \"kubernetes.io/projected/d74a1be9-254d-4ec7-a66a-e521e4d19dca-kube-api-access-ltx8m\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:51.998632 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998503 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-node-log\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.998632 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-cni-bin\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.998632 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.998862 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b-konnectivity-ca\") pod \"konnectivity-agent-mnprz\" (UID: \"33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b\") " pod="kube-system/konnectivity-agent-mnprz" Jul 10 02:22:51.998862 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998662 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d74a1be9-254d-4ec7-a66a-e521e4d19dca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:51.998862 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998682 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d74a1be9-254d-4ec7-a66a-e521e4d19dca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:51.998862 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998712 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-slash\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.998862 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998729 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82138b66-c754-4c2d-a828-a104a92a9968-kubelet-config\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:51.998862 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998783 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-kubelet\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.998862 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998811 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-run-netns\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.998862 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998835 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mrv4l" Jul 10 02:22:51.998862 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-cni-netd\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.998862 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998864 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61241733-b531-4151-a869-cc89000e2e99-env-overrides\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.999384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44vph\" (UniqueName: \"kubernetes.io/projected/61241733-b531-4151-a869-cc89000e2e99-kube-api-access-44vph\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.999384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998910 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b-agent-certs\") pod \"konnectivity-agent-mnprz\" (UID: \"33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b\") " pod="kube-system/konnectivity-agent-mnprz" Jul 10 02:22:51.999384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.998952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d74a1be9-254d-4ec7-a66a-e521e4d19dca-system-cni-dir\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:51.999384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.999032 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d74a1be9-254d-4ec7-a66a-e521e4d19dca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:51.999384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.999062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-run-openvswitch\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.999384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.999086 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-run-ovn\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.999384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.999121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61241733-b531-4151-a869-cc89000e2e99-ovn-node-metrics-cert\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.999384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.999143 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d74a1be9-254d-4ec7-a66a-e521e4d19dca-cnibin\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:51.999384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.999157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d74a1be9-254d-4ec7-a66a-e521e4d19dca-os-release\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:51.999384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.999172 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-systemd-units\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.999384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.999193 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-run-systemd\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:51.999384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:51.999236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-etc-openvswitch\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.001754 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.000613 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.001754 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.001572 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 02:22:52.001754 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.001594 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 02:22:52.001960 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.001857 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:22:52.002099 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.002079 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qlpg9\"" Jul 10 02:22:52.002707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.002691 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.004493 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.004467 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 02:22:52.004493 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.004483 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d47h5" Jul 10 02:22:52.004701 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.004680 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 02:22:52.004701 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.004687 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 02:22:52.005126 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.005111 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2pl9h\"" Jul 10 02:22:52.005217 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.005198 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:22:52.005496 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.005480 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6pxzv\"" Jul 10 02:22:52.005764 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.005738 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 02:22:52.005764 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.005761 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hv4z2" Jul 10 02:22:52.006722 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.006708 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 02:22:52.007774 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.007757 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 02:22:52.007832 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.007788 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-487wq\"" Jul 10 02:22:52.009322 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.009285 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 02:22:52.009409 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.009337 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 02:22:52.009465 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.009452 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 02:22:52.009874 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.009858 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dhfx6\"" Jul 10 02:22:52.010102 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.010088 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 02:22:52.033746 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.033724 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p8m54" Jul 10 02:22:52.043070 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.043048 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p8m54" Jul 10 02:22:52.053619 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.053557 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 02:17:51 +0000 UTC" deadline="2027-12-19 13:58:58.191312429 +0000 UTC" Jul 10 02:22:52.053619 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.053616 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12659h36m6.137699993s" Jul 10 02:22:52.093079 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.093055 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 02:22:52.099420 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d74a1be9-254d-4ec7-a66a-e521e4d19dca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.099532 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099428 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-run-openvswitch\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.099532 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099449 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-run-ovn\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.099532 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjmgn\" (UniqueName: \"kubernetes.io/projected/76708d83-5234-4c4a-b588-7fba2172d21a-kube-api-access-cjmgn\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.099532 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-system-cni-dir\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.099532 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099506 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-run-openvswitch\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.099881 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-run-k8s-cni-cncf-io\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.099881 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-run-multus-certs\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.099881 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099585 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-run-ovn\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.099881 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099589 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d74a1be9-254d-4ec7-a66a-e521e4d19dca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.099881 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099587 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq2z4\" (UniqueName: \"kubernetes.io/projected/72680f75-f579-4701-a4a8-2ff305b2c047-kube-api-access-nq2z4\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.099881 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099623 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-modprobe-d\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.099881 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099662 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61241733-b531-4151-a869-cc89000e2e99-ovnkube-script-lib\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.099881 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099772 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-sysconfig\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.099881 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099807 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-run\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.099881 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099846 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d74a1be9-254d-4ec7-a66a-e521e4d19dca-cni-binary-copy\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.099881 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-var-lib-openvswitch\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.100338 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.099923 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-var-lib-openvswitch\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.100338 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100023 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-log-socket\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.100338 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100055 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-hostroot\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.100338 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100079 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-sysctl-d\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.100338 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100105 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-cni-bin\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.100338 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100120 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-log-socket\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.100338 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.100338 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100177 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.100338 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100263 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-cni-bin\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.100758 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100362 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-device-dir\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.100758 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100505 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d74a1be9-254d-4ec7-a66a-e521e4d19dca-cni-binary-copy\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.100858 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100805 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61241733-b531-4151-a869-cc89000e2e99-ovnkube-script-lib\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.100908 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100409 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2929b6d-dfa7-448e-8560-235cddb333e8-tmp-dir\") pod \"node-resolver-d47h5\" (UID: \"a2929b6d-dfa7-448e-8560-235cddb333e8\") " pod="openshift-dns/node-resolver-d47h5" Jul 10 02:22:52.100987 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100905 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b-konnectivity-ca\") pod \"konnectivity-agent-mnprz\" (UID: \"33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b\") " pod="kube-system/konnectivity-agent-mnprz" Jul 10 02:22:52.100987 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100927 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d74a1be9-254d-4ec7-a66a-e521e4d19dca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.100987 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100944 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-os-release\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.100987 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.100965 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-cni-binary-copy\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.101183 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f510e04-1bc2-4f2b-9ccd-3931b38ed667-host-slash\") pod \"iptables-alerter-mrv4l\" (UID: \"7f510e04-1bc2-4f2b-9ccd-3931b38ed667\") " pod="openshift-network-operator/iptables-alerter-mrv4l" Jul 10 02:22:52.101183 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101109 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqhmn\" (UniqueName: \"kubernetes.io/projected/90c11f6c-5762-4634-9dad-5dc5e5b751eb-kube-api-access-kqhmn\") pod \"network-metrics-daemon-r4t9z\" (UID: \"90c11f6c-5762-4634-9dad-5dc5e5b751eb\") " pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:22:52.101183 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101136 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-sysctl-conf\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.101183 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-kubelet\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.101415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101197 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmt4x\" (UniqueName: \"kubernetes.io/projected/7f510e04-1bc2-4f2b-9ccd-3931b38ed667-kube-api-access-kmt4x\") pod \"iptables-alerter-mrv4l\" (UID: \"7f510e04-1bc2-4f2b-9ccd-3931b38ed667\") " pod="openshift-network-operator/iptables-alerter-mrv4l" Jul 10 02:22:52.101415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101234 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-kubernetes\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.101415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101267 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-kubelet\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.101415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101290 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsffj\" (UniqueName: \"kubernetes.io/projected/a2929b6d-dfa7-448e-8560-235cddb333e8-kube-api-access-nsffj\") pod \"node-resolver-d47h5\" (UID: \"a2929b6d-dfa7-448e-8560-235cddb333e8\") " pod="openshift-dns/node-resolver-d47h5" Jul 10 02:22:52.101415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101320 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61241733-b531-4151-a869-cc89000e2e99-env-overrides\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.101415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44vph\" (UniqueName: \"kubernetes.io/projected/61241733-b531-4151-a869-cc89000e2e99-kube-api-access-44vph\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.101415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101366 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b-agent-certs\") pod \"konnectivity-agent-mnprz\" (UID: \"33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b\") " pod="kube-system/konnectivity-agent-mnprz" Jul 10 02:22:52.101415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d74a1be9-254d-4ec7-a66a-e521e4d19dca-system-cni-dir\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.101415 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61241733-b531-4151-a869-cc89000e2e99-ovn-node-metrics-cert\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.101843 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101436 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-multus-conf-dir\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.101843 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101455 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d74a1be9-254d-4ec7-a66a-e521e4d19dca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.101843 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101460 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk94p\" (UniqueName: \"kubernetes.io/projected/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-kube-api-access-nk94p\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.101843 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101507 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d74a1be9-254d-4ec7-a66a-e521e4d19dca-system-cni-dir\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.101843 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101560 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b-konnectivity-ca\") pod \"konnectivity-agent-mnprz\" (UID: \"33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b\") " pod="kube-system/konnectivity-agent-mnprz" Jul 10 02:22:52.101843 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101742 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61241733-b531-4151-a869-cc89000e2e99-env-overrides\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.101843 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101765 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 02:22:52.101843 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101792 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/62337cd8-e0f1-421f-9b88-e9a4f643841a-serviceca\") pod \"node-ca-hv4z2\" (UID: \"62337cd8-e0f1-421f-9b88-e9a4f643841a\") " pod="openshift-image-registry/node-ca-hv4z2" Jul 10 02:22:52.102212 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101859 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-sys-fs\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.102212 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2929b6d-dfa7-448e-8560-235cddb333e8-hosts-file\") pod \"node-resolver-d47h5\" (UID: \"a2929b6d-dfa7-448e-8560-235cddb333e8\") " pod="openshift-dns/node-resolver-d47h5" Jul 10 02:22:52.102212 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d74a1be9-254d-4ec7-a66a-e521e4d19dca-cnibin\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.102212 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.101966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d74a1be9-254d-4ec7-a66a-e521e4d19dca-os-release\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.102212 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102001 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-systemd-units\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.102212 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-run-systemd\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.102212 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102097 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-systemd-units\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.102212 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102098 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-etc-openvswitch\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.102212 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102137 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-etc-openvswitch\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.102212 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102181 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-cnibin\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.102212 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102201 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-var-lib-cni-multus\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.102668 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102309 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d74a1be9-254d-4ec7-a66a-e521e4d19dca-os-release\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.102668 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102390 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d74a1be9-254d-4ec7-a66a-e521e4d19dca-cnibin\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.102668 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102426 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-run-systemd\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.102668 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102220 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-var-lib-kubelet\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.102945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82138b66-c754-4c2d-a828-a104a92a9968-dbus\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:52.102945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102706 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-run-ovn-kubernetes\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.102945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102726 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61241733-b531-4151-a869-cc89000e2e99-ovnkube-config\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.102945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102747 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-multus-cni-dir\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.102945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102766 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-multus-socket-dir-parent\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.102945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7f510e04-1bc2-4f2b-9ccd-3931b38ed667-iptables-alerter-script\") pod \"iptables-alerter-mrv4l\" (UID: \"7f510e04-1bc2-4f2b-9ccd-3931b38ed667\") " pod="openshift-network-operator/iptables-alerter-mrv4l" Jul 10 02:22:52.102945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102791 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-run-ovn-kubernetes\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.102945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-run-netns\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.102945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnt5v\" (UniqueName: \"kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v\") pod \"network-check-target-j28gk\" (UID: \"e5e656b6-b264-455f-9da1-2de1fbac1a9f\") " pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:22:52.102945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.102945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102916 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-host\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.102945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82138b66-c754-4c2d-a828-a104a92a9968-dbus\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.102991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltx8m\" (UniqueName: \"kubernetes.io/projected/d74a1be9-254d-4ec7-a66a-e521e4d19dca-kube-api-access-ltx8m\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-node-log\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103038 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-multus-daemon-config\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103059 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62337cd8-e0f1-421f-9b88-e9a4f643841a-host\") pod \"node-ca-hv4z2\" (UID: \"62337cd8-e0f1-421f-9b88-e9a4f643841a\") " pod="openshift-image-registry/node-ca-hv4z2" Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103083 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-socket-dir\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103086 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-node-log\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103105 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-registration-dir\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103139 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d74a1be9-254d-4ec7-a66a-e521e4d19dca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.103149 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.103211 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret podName:82138b66-c754-4c2d-a828-a104a92a9968 nodeName:}" failed. No retries permitted until 2026-07-10 02:22:52.603192209 +0000 UTC m=+2.002645112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret") pod "global-pull-secret-syncer-lgw26" (UID: "82138b66-c754-4c2d-a828-a104a92a9968") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103262 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61241733-b531-4151-a869-cc89000e2e99-ovnkube-config\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-slash\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-var-lib-kubelet\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103417 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-slash\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.103674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103514 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-sys\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103538 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-lib-modules\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103559 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76708d83-5234-4c4a-b588-7fba2172d21a-tmp\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103587 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82138b66-c754-4c2d-a828-a104a92a9968-kubelet-config\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vv4l\" (UniqueName: \"kubernetes.io/projected/62337cd8-e0f1-421f-9b88-e9a4f643841a-kube-api-access-2vv4l\") pod \"node-ca-hv4z2\" (UID: \"62337cd8-e0f1-421f-9b88-e9a4f643841a\") " pod="openshift-image-registry/node-ca-hv4z2" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-run-netns\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103662 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-cni-netd\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103683 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82138b66-c754-4c2d-a828-a104a92a9968-kubelet-config\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-var-lib-cni-bin\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103735 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-cni-netd\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103765 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-etc-kubernetes\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103786 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61241733-b531-4151-a869-cc89000e2e99-host-run-netns\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103792 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs\") pod \"network-metrics-daemon-r4t9z\" (UID: \"90c11f6c-5762-4634-9dad-5dc5e5b751eb\") " pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103816 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-systemd\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.103838 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76708d83-5234-4c4a-b588-7fba2172d21a-etc-tuned\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.104240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.104233 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d74a1be9-254d-4ec7-a66a-e521e4d19dca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.105261 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.105228 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61241733-b531-4151-a869-cc89000e2e99-ovn-node-metrics-cert\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.105372 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.105357 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b-agent-certs\") pod \"konnectivity-agent-mnprz\" (UID: \"33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b\") " pod="kube-system/konnectivity-agent-mnprz" Jul 10 02:22:52.112791 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:52.112740 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174a0477fbdfffd07715b9db16c916ce.slice/crio-64c8dd4904578158fb9b411d67bcba84c53a5ce5621af32646b4b37982b4dfeb WatchSource:0}: Error finding container 64c8dd4904578158fb9b411d67bcba84c53a5ce5621af32646b4b37982b4dfeb: Status 404 returned error can't find the container with id 64c8dd4904578158fb9b411d67bcba84c53a5ce5621af32646b4b37982b4dfeb Jul 10 02:22:52.113288 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.113262 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44vph\" (UniqueName: \"kubernetes.io/projected/61241733-b531-4151-a869-cc89000e2e99-kube-api-access-44vph\") pod \"ovnkube-node-nlm7j\" (UID: \"61241733-b531-4151-a869-cc89000e2e99\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.113487 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:52.113458 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042669854574a08b037a614d9f1b2909.slice/crio-95132b2fd87ce0ac57017922070276015fb11b2a44f82f8f1884aa88bdf076f0 WatchSource:0}: Error finding container 95132b2fd87ce0ac57017922070276015fb11b2a44f82f8f1884aa88bdf076f0: Status 404 returned error can't find the container with id 95132b2fd87ce0ac57017922070276015fb11b2a44f82f8f1884aa88bdf076f0 Jul 10 02:22:52.115024 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.115001 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltx8m\" (UniqueName: \"kubernetes.io/projected/d74a1be9-254d-4ec7-a66a-e521e4d19dca-kube-api-access-ltx8m\") pod \"multus-additional-cni-plugins-sd54c\" (UID: \"d74a1be9-254d-4ec7-a66a-e521e4d19dca\") " pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.121093 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.121076 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 02:22:52.126427 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.126385 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.ec2.internal" event={"ID":"042669854574a08b037a614d9f1b2909","Type":"ContainerStarted","Data":"95132b2fd87ce0ac57017922070276015fb11b2a44f82f8f1884aa88bdf076f0"} Jul 10 02:22:52.127655 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.127634 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal" event={"ID":"174a0477fbdfffd07715b9db16c916ce","Type":"ContainerStarted","Data":"64c8dd4904578158fb9b411d67bcba84c53a5ce5621af32646b4b37982b4dfeb"} Jul 10 02:22:52.204574 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204541 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-hostroot\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.204574 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-sysctl-d\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.204776 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-device-dir\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.204776 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204613 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2929b6d-dfa7-448e-8560-235cddb333e8-tmp-dir\") pod \"node-resolver-d47h5\" (UID: \"a2929b6d-dfa7-448e-8560-235cddb333e8\") " pod="openshift-dns/node-resolver-d47h5" Jul 10 02:22:52.204776 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204686 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-hostroot\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.204776 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204692 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-os-release\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.204776 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-cni-binary-copy\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.204776 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f510e04-1bc2-4f2b-9ccd-3931b38ed667-host-slash\") pod \"iptables-alerter-mrv4l\" (UID: \"7f510e04-1bc2-4f2b-9ccd-3931b38ed667\") " pod="openshift-network-operator/iptables-alerter-mrv4l" Jul 10 02:22:52.204776 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204765 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-sysctl-d\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqhmn\" (UniqueName: \"kubernetes.io/projected/90c11f6c-5762-4634-9dad-5dc5e5b751eb-kube-api-access-kqhmn\") pod \"network-metrics-daemon-r4t9z\" (UID: \"90c11f6c-5762-4634-9dad-5dc5e5b751eb\") " pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204805 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-os-release\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204832 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f510e04-1bc2-4f2b-9ccd-3931b38ed667-host-slash\") pod \"iptables-alerter-mrv4l\" (UID: \"7f510e04-1bc2-4f2b-9ccd-3931b38ed667\") " pod="openshift-network-operator/iptables-alerter-mrv4l" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-sysctl-conf\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmt4x\" (UniqueName: \"kubernetes.io/projected/7f510e04-1bc2-4f2b-9ccd-3931b38ed667-kube-api-access-kmt4x\") pod \"iptables-alerter-mrv4l\" (UID: \"7f510e04-1bc2-4f2b-9ccd-3931b38ed667\") " pod="openshift-network-operator/iptables-alerter-mrv4l" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204915 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2929b6d-dfa7-448e-8560-235cddb333e8-tmp-dir\") pod \"node-resolver-d47h5\" (UID: \"a2929b6d-dfa7-448e-8560-235cddb333e8\") " pod="openshift-dns/node-resolver-d47h5" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204966 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-kubernetes\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-kubernetes\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.204997 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsffj\" (UniqueName: \"kubernetes.io/projected/a2929b6d-dfa7-448e-8560-235cddb333e8-kube-api-access-nsffj\") pod \"node-resolver-d47h5\" (UID: \"a2929b6d-dfa7-448e-8560-235cddb333e8\") " pod="openshift-dns/node-resolver-d47h5" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205015 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-sysctl-conf\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-multus-conf-dir\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk94p\" (UniqueName: \"kubernetes.io/projected/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-kube-api-access-nk94p\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/62337cd8-e0f1-421f-9b88-e9a4f643841a-serviceca\") pod \"node-ca-hv4z2\" (UID: \"62337cd8-e0f1-421f-9b88-e9a4f643841a\") " pod="openshift-image-registry/node-ca-hv4z2" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205097 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-multus-conf-dir\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-sys-fs\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205158 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-sys-fs\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.205159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2929b6d-dfa7-448e-8560-235cddb333e8-hosts-file\") pod \"node-resolver-d47h5\" (UID: \"a2929b6d-dfa7-448e-8560-235cddb333e8\") " pod="openshift-dns/node-resolver-d47h5" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205202 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-cnibin\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205210 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-device-dir\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-var-lib-cni-multus\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-var-lib-kubelet\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-multus-cni-dir\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205290 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-cnibin\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-multus-socket-dir-parent\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205324 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2929b6d-dfa7-448e-8560-235cddb333e8-hosts-file\") pod \"node-resolver-d47h5\" (UID: \"a2929b6d-dfa7-448e-8560-235cddb333e8\") " pod="openshift-dns/node-resolver-d47h5" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205337 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-var-lib-kubelet\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205296 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-var-lib-cni-multus\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205357 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-multus-cni-dir\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205328 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7f510e04-1bc2-4f2b-9ccd-3931b38ed667-iptables-alerter-script\") pod \"iptables-alerter-mrv4l\" (UID: \"7f510e04-1bc2-4f2b-9ccd-3931b38ed667\") " pod="openshift-network-operator/iptables-alerter-mrv4l" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205373 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-cni-binary-copy\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205376 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-multus-socket-dir-parent\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205399 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-run-netns\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnt5v\" (UniqueName: \"kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v\") pod \"network-check-target-j28gk\" (UID: \"e5e656b6-b264-455f-9da1-2de1fbac1a9f\") " pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205421 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-run-netns\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.207866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-host\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-host\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-multus-daemon-config\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62337cd8-e0f1-421f-9b88-e9a4f643841a-host\") pod \"node-ca-hv4z2\" (UID: \"62337cd8-e0f1-421f-9b88-e9a4f643841a\") " pod="openshift-image-registry/node-ca-hv4z2" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205573 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205580 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-socket-dir\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205607 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/62337cd8-e0f1-421f-9b88-e9a4f643841a-serviceca\") pod \"node-ca-hv4z2\" (UID: \"62337cd8-e0f1-421f-9b88-e9a4f643841a\") " pod="openshift-image-registry/node-ca-hv4z2" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205621 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-registration-dir\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205631 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62337cd8-e0f1-421f-9b88-e9a4f643841a-host\") pod \"node-ca-hv4z2\" (UID: \"62337cd8-e0f1-421f-9b88-e9a4f643841a\") " pod="openshift-image-registry/node-ca-hv4z2" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-var-lib-kubelet\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205678 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-registration-dir\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205682 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-socket-dir\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205688 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-var-lib-kubelet\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205691 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-sys\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205730 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72680f75-f579-4701-a4a8-2ff305b2c047-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205751 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-lib-modules\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.208529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205772 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-sys\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76708d83-5234-4c4a-b588-7fba2172d21a-tmp\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vv4l\" (UniqueName: \"kubernetes.io/projected/62337cd8-e0f1-421f-9b88-e9a4f643841a-kube-api-access-2vv4l\") pod \"node-ca-hv4z2\" (UID: \"62337cd8-e0f1-421f-9b88-e9a4f643841a\") " pod="openshift-image-registry/node-ca-hv4z2" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205827 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7f510e04-1bc2-4f2b-9ccd-3931b38ed667-iptables-alerter-script\") pod \"iptables-alerter-mrv4l\" (UID: \"7f510e04-1bc2-4f2b-9ccd-3931b38ed667\") " pod="openshift-network-operator/iptables-alerter-mrv4l" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-var-lib-cni-bin\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205866 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-var-lib-cni-bin\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-etc-kubernetes\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205881 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-lib-modules\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205903 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs\") pod \"network-metrics-daemon-r4t9z\" (UID: \"90c11f6c-5762-4634-9dad-5dc5e5b751eb\") " pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205919 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-etc-kubernetes\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-systemd\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.205981 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-systemd\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.206043 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76708d83-5234-4c4a-b588-7fba2172d21a-etc-tuned\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.206076 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjmgn\" (UniqueName: \"kubernetes.io/projected/76708d83-5234-4c4a-b588-7fba2172d21a-kube-api-access-cjmgn\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.206098 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-system-cni-dir\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.206118 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-run-k8s-cni-cncf-io\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.206178 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-run-k8s-cni-cncf-io\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.206186 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-system-cni-dir\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.209008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.206320 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-run-multus-certs\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.209550 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.206203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-host-run-multus-certs\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.209550 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.207576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq2z4\" (UniqueName: \"kubernetes.io/projected/72680f75-f579-4701-a4a8-2ff305b2c047-kube-api-access-nq2z4\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.209550 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.207614 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:22:52.209550 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.207678 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs podName:90c11f6c-5762-4634-9dad-5dc5e5b751eb nodeName:}" failed. No retries permitted until 2026-07-10 02:22:52.707660565 +0000 UTC m=+2.107113477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs") pod "network-metrics-daemon-r4t9z" (UID: "90c11f6c-5762-4634-9dad-5dc5e5b751eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:22:52.209550 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.207714 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-modprobe-d\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.209550 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.207616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-modprobe-d\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.209550 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.207769 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-sysconfig\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.209550 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.207808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-run\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.209550 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.207879 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-etc-sysconfig\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.209550 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.207884 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76708d83-5234-4c4a-b588-7fba2172d21a-run\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.209550 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.207943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-multus-daemon-config\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.209550 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.208025 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76708d83-5234-4c4a-b588-7fba2172d21a-etc-tuned\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.209550 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.208233 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76708d83-5234-4c4a-b588-7fba2172d21a-tmp\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.212552 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.212532 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:22:52.212552 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.212552 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:22:52.212718 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.212562 2575 projected.go:196] Error preparing data for projected volume kube-api-access-tnt5v for pod openshift-network-diagnostics/network-check-target-j28gk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:22:52.212718 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.212618 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v podName:e5e656b6-b264-455f-9da1-2de1fbac1a9f nodeName:}" failed. No retries permitted until 2026-07-10 02:22:52.712601606 +0000 UTC m=+2.112054518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tnt5v" (UniqueName: "kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v") pod "network-check-target-j28gk" (UID: "e5e656b6-b264-455f-9da1-2de1fbac1a9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:22:52.214843 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.214823 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsffj\" (UniqueName: \"kubernetes.io/projected/a2929b6d-dfa7-448e-8560-235cddb333e8-kube-api-access-nsffj\") pod \"node-resolver-d47h5\" (UID: \"a2929b6d-dfa7-448e-8560-235cddb333e8\") " pod="openshift-dns/node-resolver-d47h5" Jul 10 02:22:52.214911 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.214896 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqhmn\" (UniqueName: \"kubernetes.io/projected/90c11f6c-5762-4634-9dad-5dc5e5b751eb-kube-api-access-kqhmn\") pod \"network-metrics-daemon-r4t9z\" (UID: \"90c11f6c-5762-4634-9dad-5dc5e5b751eb\") " pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:22:52.214970 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.214948 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmt4x\" (UniqueName: \"kubernetes.io/projected/7f510e04-1bc2-4f2b-9ccd-3931b38ed667-kube-api-access-kmt4x\") pod \"iptables-alerter-mrv4l\" (UID: \"7f510e04-1bc2-4f2b-9ccd-3931b38ed667\") " pod="openshift-network-operator/iptables-alerter-mrv4l" Jul 10 02:22:52.215083 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.215064 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk94p\" (UniqueName: \"kubernetes.io/projected/3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee-kube-api-access-nk94p\") pod \"multus-blpm7\" (UID: \"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee\") " pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.215309 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.215290 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjmgn\" (UniqueName: \"kubernetes.io/projected/76708d83-5234-4c4a-b588-7fba2172d21a-kube-api-access-cjmgn\") pod \"tuned-gxxlp\" (UID: \"76708d83-5234-4c4a-b588-7fba2172d21a\") " pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.215677 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.215660 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vv4l\" (UniqueName: \"kubernetes.io/projected/62337cd8-e0f1-421f-9b88-e9a4f643841a-kube-api-access-2vv4l\") pod \"node-ca-hv4z2\" (UID: \"62337cd8-e0f1-421f-9b88-e9a4f643841a\") " pod="openshift-image-registry/node-ca-hv4z2" Jul 10 02:22:52.217524 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.217479 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq2z4\" (UniqueName: \"kubernetes.io/projected/72680f75-f579-4701-a4a8-2ff305b2c047-kube-api-access-nq2z4\") pod \"aws-ebs-csi-driver-node-5tbtt\" (UID: \"72680f75-f579-4701-a4a8-2ff305b2c047\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.329088 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.329063 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sd54c" Jul 10 02:22:52.337713 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:52.337690 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74a1be9_254d_4ec7_a66a_e521e4d19dca.slice/crio-2f1bf088588534c7e331193ad0d01f8c9896c1a24ae40da3a65be5ee69a40ba5 WatchSource:0}: Error finding container 2f1bf088588534c7e331193ad0d01f8c9896c1a24ae40da3a65be5ee69a40ba5: Status 404 returned error can't find the container with id 2f1bf088588534c7e331193ad0d01f8c9896c1a24ae40da3a65be5ee69a40ba5 Jul 10 02:22:52.344725 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.344708 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:22:52.351372 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:52.351347 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61241733_b531_4151_a869_cc89000e2e99.slice/crio-9bf59c7e73b752c3eaddffeb5c138b755cbcc95666a6d6d97e6705fb894e508c WatchSource:0}: Error finding container 9bf59c7e73b752c3eaddffeb5c138b755cbcc95666a6d6d97e6705fb894e508c: Status 404 returned error can't find the container with id 9bf59c7e73b752c3eaddffeb5c138b755cbcc95666a6d6d97e6705fb894e508c Jul 10 02:22:52.356703 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.356689 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mnprz" Jul 10 02:22:52.366204 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:52.366179 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33cbbbcf_226b_4c91_8f18_f3e5ab0ee24b.slice/crio-65a021ba8538af631496fa917860c1cc37f979355df373b3ad8bb3e4571d6b50 WatchSource:0}: Error finding container 65a021ba8538af631496fa917860c1cc37f979355df373b3ad8bb3e4571d6b50: Status 404 returned error can't find the container with id 65a021ba8538af631496fa917860c1cc37f979355df373b3ad8bb3e4571d6b50 Jul 10 02:22:52.370956 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.370939 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-blpm7" Jul 10 02:22:52.377615 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:52.377591 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed8f1f1_c480_41a9_b2f6_2bfabe74c7ee.slice/crio-951e480c5aeca37fcb2dc0e1544aa008389cae0c4c3d5a93e0ea3afb05f605d0 WatchSource:0}: Error finding container 951e480c5aeca37fcb2dc0e1544aa008389cae0c4c3d5a93e0ea3afb05f605d0: Status 404 returned error can't find the container with id 951e480c5aeca37fcb2dc0e1544aa008389cae0c4c3d5a93e0ea3afb05f605d0 Jul 10 02:22:52.395672 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.395651 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mrv4l" Jul 10 02:22:52.402302 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:52.402277 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f510e04_1bc2_4f2b_9ccd_3931b38ed667.slice/crio-a26a45427869abf1275ead585108e7364edd9a02c3608e4ab81ea78f084912e4 WatchSource:0}: Error finding container a26a45427869abf1275ead585108e7364edd9a02c3608e4ab81ea78f084912e4: Status 404 returned error can't find the container with id a26a45427869abf1275ead585108e7364edd9a02c3608e4ab81ea78f084912e4 Jul 10 02:22:52.411274 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.411239 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" Jul 10 02:22:52.418163 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:52.418143 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72680f75_f579_4701_a4a8_2ff305b2c047.slice/crio-b8dfc3cd4c8c38e512673656b32e36af18da0a7b7e4355edd863a33717380a53 WatchSource:0}: Error finding container b8dfc3cd4c8c38e512673656b32e36af18da0a7b7e4355edd863a33717380a53: Status 404 returned error can't find the container with id b8dfc3cd4c8c38e512673656b32e36af18da0a7b7e4355edd863a33717380a53 Jul 10 02:22:52.423076 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.423060 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" Jul 10 02:22:52.429030 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.429011 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d47h5" Jul 10 02:22:52.429982 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:52.429960 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76708d83_5234_4c4a_b588_7fba2172d21a.slice/crio-02e4bd60d790721659be795e57a710f57b45c66517b29df6d845703daade2ab0 WatchSource:0}: Error finding container 02e4bd60d790721659be795e57a710f57b45c66517b29df6d845703daade2ab0: Status 404 returned error can't find the container with id 02e4bd60d790721659be795e57a710f57b45c66517b29df6d845703daade2ab0 Jul 10 02:22:52.433559 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.433545 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hv4z2" Jul 10 02:22:52.436724 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:52.436707 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2929b6d_dfa7_448e_8560_235cddb333e8.slice/crio-d25e4bb3aa21984194f96ea2092b61752dfba30326656619e653a1611984f037 WatchSource:0}: Error finding container d25e4bb3aa21984194f96ea2092b61752dfba30326656619e653a1611984f037: Status 404 returned error can't find the container with id d25e4bb3aa21984194f96ea2092b61752dfba30326656619e653a1611984f037 Jul 10 02:22:52.440125 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:22:52.440107 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62337cd8_e0f1_421f_9b88_e9a4f643841a.slice/crio-547bacfdcf162bbd6d4afdc29067ef3e6a0d26b6aeaa66bd7c72d8a50b67e139 WatchSource:0}: Error finding container 547bacfdcf162bbd6d4afdc29067ef3e6a0d26b6aeaa66bd7c72d8a50b67e139: Status 404 returned error can't find the container with id 547bacfdcf162bbd6d4afdc29067ef3e6a0d26b6aeaa66bd7c72d8a50b67e139 Jul 10 02:22:52.476998 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.476948 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:22:52.610793 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.610755 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:52.610964 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.610930 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:22:52.611027 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.611002 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret podName:82138b66-c754-4c2d-a828-a104a92a9968 nodeName:}" failed. No retries permitted until 2026-07-10 02:22:53.610983341 +0000 UTC m=+3.010436234 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret") pod "global-pull-secret-syncer-lgw26" (UID: "82138b66-c754-4c2d-a828-a104a92a9968") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:22:52.711800 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.711765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs\") pod \"network-metrics-daemon-r4t9z\" (UID: \"90c11f6c-5762-4634-9dad-5dc5e5b751eb\") " pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:22:52.711962 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.711887 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:22:52.711962 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.711960 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs podName:90c11f6c-5762-4634-9dad-5dc5e5b751eb nodeName:}" failed. No retries permitted until 2026-07-10 02:22:53.711939312 +0000 UTC m=+3.111392220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs") pod "network-metrics-daemon-r4t9z" (UID: "90c11f6c-5762-4634-9dad-5dc5e5b751eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:22:52.812858 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:52.812195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnt5v\" (UniqueName: \"kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v\") pod \"network-check-target-j28gk\" (UID: \"e5e656b6-b264-455f-9da1-2de1fbac1a9f\") " pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:22:52.812858 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.812392 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:22:52.812858 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.812412 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:22:52.812858 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.812425 2575 projected.go:196] Error preparing data for projected volume kube-api-access-tnt5v for pod openshift-network-diagnostics/network-check-target-j28gk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:22:52.812858 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:52.812479 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v podName:e5e656b6-b264-455f-9da1-2de1fbac1a9f nodeName:}" failed. No retries permitted until 2026-07-10 02:22:53.812460812 +0000 UTC m=+3.211913709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnt5v" (UniqueName: "kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v") pod "network-check-target-j28gk" (UID: "e5e656b6-b264-455f-9da1-2de1fbac1a9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:22:53.044845 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.044786 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 02:17:52 +0000 UTC" deadline="2028-03-31 11:25:50.068890142 +0000 UTC" Jul 10 02:22:53.044845 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.044843 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15129h2m57.024050413s" Jul 10 02:22:53.154104 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.154020 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hv4z2" event={"ID":"62337cd8-e0f1-421f-9b88-e9a4f643841a","Type":"ContainerStarted","Data":"547bacfdcf162bbd6d4afdc29067ef3e6a0d26b6aeaa66bd7c72d8a50b67e139"} Jul 10 02:22:53.172948 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.172895 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" event={"ID":"76708d83-5234-4c4a-b588-7fba2172d21a","Type":"ContainerStarted","Data":"02e4bd60d790721659be795e57a710f57b45c66517b29df6d845703daade2ab0"} Jul 10 02:22:53.193930 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.193825 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" event={"ID":"72680f75-f579-4701-a4a8-2ff305b2c047","Type":"ContainerStarted","Data":"b8dfc3cd4c8c38e512673656b32e36af18da0a7b7e4355edd863a33717380a53"} Jul 10 02:22:53.213554 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.213467 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mrv4l" event={"ID":"7f510e04-1bc2-4f2b-9ccd-3931b38ed667","Type":"ContainerStarted","Data":"a26a45427869abf1275ead585108e7364edd9a02c3608e4ab81ea78f084912e4"} Jul 10 02:22:53.221927 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.221897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-blpm7" event={"ID":"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee","Type":"ContainerStarted","Data":"951e480c5aeca37fcb2dc0e1544aa008389cae0c4c3d5a93e0ea3afb05f605d0"} Jul 10 02:22:53.226297 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.226230 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mnprz" event={"ID":"33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b","Type":"ContainerStarted","Data":"65a021ba8538af631496fa917860c1cc37f979355df373b3ad8bb3e4571d6b50"} Jul 10 02:22:53.235066 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.235001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d47h5" event={"ID":"a2929b6d-dfa7-448e-8560-235cddb333e8","Type":"ContainerStarted","Data":"d25e4bb3aa21984194f96ea2092b61752dfba30326656619e653a1611984f037"} Jul 10 02:22:53.243595 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.243229 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" event={"ID":"61241733-b531-4151-a869-cc89000e2e99","Type":"ContainerStarted","Data":"9bf59c7e73b752c3eaddffeb5c138b755cbcc95666a6d6d97e6705fb894e508c"} Jul 10 02:22:53.257671 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.257619 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd54c" event={"ID":"d74a1be9-254d-4ec7-a66a-e521e4d19dca","Type":"ContainerStarted","Data":"2f1bf088588534c7e331193ad0d01f8c9896c1a24ae40da3a65be5ee69a40ba5"} Jul 10 02:22:53.407906 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.407831 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:22:53.620904 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.620823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:53.621057 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:53.620962 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:22:53.621057 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:53.621049 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret podName:82138b66-c754-4c2d-a828-a104a92a9968 nodeName:}" failed. No retries permitted until 2026-07-10 02:22:55.621029529 +0000 UTC m=+5.020482427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret") pod "global-pull-secret-syncer-lgw26" (UID: "82138b66-c754-4c2d-a828-a104a92a9968") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:22:53.722053 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.721970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs\") pod \"network-metrics-daemon-r4t9z\" (UID: \"90c11f6c-5762-4634-9dad-5dc5e5b751eb\") " pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:22:53.722208 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:53.722134 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:22:53.722208 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:53.722194 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs podName:90c11f6c-5762-4634-9dad-5dc5e5b751eb nodeName:}" failed. No retries permitted until 2026-07-10 02:22:55.722176434 +0000 UTC m=+5.121629330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs") pod "network-metrics-daemon-r4t9z" (UID: "90c11f6c-5762-4634-9dad-5dc5e5b751eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:22:53.823882 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:53.823847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnt5v\" (UniqueName: \"kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v\") pod \"network-check-target-j28gk\" (UID: \"e5e656b6-b264-455f-9da1-2de1fbac1a9f\") " pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:22:53.824054 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:53.824022 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:22:53.824054 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:53.824041 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:22:53.824054 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:53.824054 2575 projected.go:196] Error preparing data for projected volume kube-api-access-tnt5v for pod openshift-network-diagnostics/network-check-target-j28gk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:22:53.824233 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:53.824111 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v podName:e5e656b6-b264-455f-9da1-2de1fbac1a9f nodeName:}" failed. No retries permitted until 2026-07-10 02:22:55.82409248 +0000 UTC m=+5.223545389 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnt5v" (UniqueName: "kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v") pod "network-check-target-j28gk" (UID: "e5e656b6-b264-455f-9da1-2de1fbac1a9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:22:54.046106 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:54.045990 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 02:17:52 +0000 UTC" deadline="2027-12-15 20:35:01.998110266 +0000 UTC" Jul 10 02:22:54.046106 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:54.046029 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12570h12m7.95208627s" Jul 10 02:22:54.123444 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:54.123402 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:22:54.123625 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:54.123526 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:22:54.123987 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:54.123911 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:54.124070 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:54.124022 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:22:54.124139 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:54.124087 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:22:54.124188 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:54.124163 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:22:54.401807 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:54.401730 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:22:54.420366 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:54.420308 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:22:55.644384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:55.644347 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:55.644911 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:55.644554 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:22:55.644911 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:55.644619 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret podName:82138b66-c754-4c2d-a828-a104a92a9968 nodeName:}" failed. No retries permitted until 2026-07-10 02:22:59.644598689 +0000 UTC m=+9.044051594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret") pod "global-pull-secret-syncer-lgw26" (UID: "82138b66-c754-4c2d-a828-a104a92a9968") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:22:55.745539 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:55.745493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs\") pod \"network-metrics-daemon-r4t9z\" (UID: \"90c11f6c-5762-4634-9dad-5dc5e5b751eb\") " pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:22:55.745762 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:55.745632 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:22:55.745762 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:55.745706 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs podName:90c11f6c-5762-4634-9dad-5dc5e5b751eb nodeName:}" failed. No retries permitted until 2026-07-10 02:22:59.745685824 +0000 UTC m=+9.145138715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs") pod "network-metrics-daemon-r4t9z" (UID: "90c11f6c-5762-4634-9dad-5dc5e5b751eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:22:55.845852 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:55.845816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnt5v\" (UniqueName: \"kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v\") pod \"network-check-target-j28gk\" (UID: \"e5e656b6-b264-455f-9da1-2de1fbac1a9f\") " pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:22:55.846046 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:55.846003 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:22:55.846046 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:55.846020 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:22:55.846046 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:55.846029 2575 projected.go:196] Error preparing data for projected volume kube-api-access-tnt5v for pod openshift-network-diagnostics/network-check-target-j28gk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:22:55.846201 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:55.846082 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v podName:e5e656b6-b264-455f-9da1-2de1fbac1a9f nodeName:}" failed. No retries permitted until 2026-07-10 02:22:59.846062585 +0000 UTC m=+9.245515476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnt5v" (UniqueName: "kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v") pod "network-check-target-j28gk" (UID: "e5e656b6-b264-455f-9da1-2de1fbac1a9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:22:56.123462 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:56.123424 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:22:56.123620 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:56.123545 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:56.123620 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:56.123556 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:22:56.123747 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:56.123660 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:22:56.123747 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:56.123726 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:22:56.123857 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:56.123811 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:22:58.123940 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:58.123427 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:58.123940 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:58.123572 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:22:58.123940 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:58.123663 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:22:58.123940 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:58.123755 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:22:58.123940 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:58.123783 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:22:58.123940 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:58.123853 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:22:59.679095 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:59.679055 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:22:59.679563 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:59.679179 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:22:59.679563 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:59.679240 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret podName:82138b66-c754-4c2d-a828-a104a92a9968 nodeName:}" failed. No retries permitted until 2026-07-10 02:23:07.679223524 +0000 UTC m=+17.078676419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret") pod "global-pull-secret-syncer-lgw26" (UID: "82138b66-c754-4c2d-a828-a104a92a9968") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:22:59.779716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:59.779490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs\") pod \"network-metrics-daemon-r4t9z\" (UID: \"90c11f6c-5762-4634-9dad-5dc5e5b751eb\") " pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:22:59.779716 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:59.779615 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:22:59.779716 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:59.779670 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs podName:90c11f6c-5762-4634-9dad-5dc5e5b751eb nodeName:}" failed. No retries permitted until 2026-07-10 02:23:07.77965244 +0000 UTC m=+17.179105334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs") pod "network-metrics-daemon-r4t9z" (UID: "90c11f6c-5762-4634-9dad-5dc5e5b751eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:22:59.880641 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:22:59.880564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnt5v\" (UniqueName: \"kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v\") pod \"network-check-target-j28gk\" (UID: \"e5e656b6-b264-455f-9da1-2de1fbac1a9f\") " pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:22:59.880803 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:59.880732 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:22:59.880803 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:59.880751 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:22:59.880803 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:59.880763 2575 projected.go:196] Error preparing data for projected volume kube-api-access-tnt5v for pod openshift-network-diagnostics/network-check-target-j28gk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:22:59.880919 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:22:59.880821 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v podName:e5e656b6-b264-455f-9da1-2de1fbac1a9f nodeName:}" failed. No retries permitted until 2026-07-10 02:23:07.880803369 +0000 UTC m=+17.280256262 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnt5v" (UniqueName: "kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v") pod "network-check-target-j28gk" (UID: "e5e656b6-b264-455f-9da1-2de1fbac1a9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:23:00.123854 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:00.123821 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:00.124021 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:00.123939 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:23:00.124125 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:00.124102 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:00.124278 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:00.124227 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:23:00.124356 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:00.124296 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:00.124406 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:00.124376 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:23:02.123626 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:02.123388 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:02.124090 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:02.123735 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:23:02.124090 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:02.123435 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:02.124090 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:02.123475 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:02.124090 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:02.123841 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:23:02.124090 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:02.123992 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:23:04.123805 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:04.123728 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:04.123805 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:04.123780 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:04.124278 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:04.123735 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:04.124278 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:04.123862 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:23:04.124278 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:04.123966 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:23:04.124278 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:04.124075 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:23:06.123845 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:06.123811 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:06.124322 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:06.123918 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:06.124322 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:06.123926 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:23:06.124322 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:06.123950 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:06.124322 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:06.124036 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:23:06.124322 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:06.124122 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:23:07.740175 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:07.740142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:07.740499 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:07.740285 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:23:07.740499 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:07.740350 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret podName:82138b66-c754-4c2d-a828-a104a92a9968 nodeName:}" failed. No retries permitted until 2026-07-10 02:23:23.740332143 +0000 UTC m=+33.139785044 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret") pod "global-pull-secret-syncer-lgw26" (UID: "82138b66-c754-4c2d-a828-a104a92a9968") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:23:07.840791 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:07.840761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs\") pod \"network-metrics-daemon-r4t9z\" (UID: \"90c11f6c-5762-4634-9dad-5dc5e5b751eb\") " pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:07.840962 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:07.840933 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:23:07.841009 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:07.841000 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs podName:90c11f6c-5762-4634-9dad-5dc5e5b751eb nodeName:}" failed. No retries permitted until 2026-07-10 02:23:23.840978206 +0000 UTC m=+33.240431097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs") pod "network-metrics-daemon-r4t9z" (UID: "90c11f6c-5762-4634-9dad-5dc5e5b751eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:23:07.941599 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:07.941565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnt5v\" (UniqueName: \"kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v\") pod \"network-check-target-j28gk\" (UID: \"e5e656b6-b264-455f-9da1-2de1fbac1a9f\") " pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:07.944200 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:07.941999 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:23:07.944200 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:07.942031 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:23:07.944200 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:07.942045 2575 projected.go:196] Error preparing data for projected volume kube-api-access-tnt5v for pod openshift-network-diagnostics/network-check-target-j28gk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:23:07.944200 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:07.942107 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v podName:e5e656b6-b264-455f-9da1-2de1fbac1a9f nodeName:}" failed. No retries permitted until 2026-07-10 02:23:23.942090164 +0000 UTC m=+33.341543070 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnt5v" (UniqueName: "kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v") pod "network-check-target-j28gk" (UID: "e5e656b6-b264-455f-9da1-2de1fbac1a9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:23:08.123997 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:08.123967 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:08.124141 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:08.124015 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:08.124141 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:08.124082 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:08.124141 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:08.124093 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:23:08.124336 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:08.124186 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:23:08.124336 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:08.124272 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:23:10.124089 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:10.124056 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:10.124635 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:10.124177 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:10.124635 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:10.124185 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:23:10.124635 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:10.124240 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:10.124635 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:10.124348 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:23:10.124635 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:10.124441 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:23:12.123514 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:12.123475 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:12.123514 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:12.123503 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:12.123987 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:12.123592 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:23:12.123987 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:12.123657 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:12.123987 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:12.123791 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:23:12.123987 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:12.123883 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:23:13.304765 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.304541 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.ec2.internal" event={"ID":"042669854574a08b037a614d9f1b2909","Type":"ContainerStarted","Data":"13a008a4901039e3e4a6982ab5bf46f51871d55252f1e4299f87db322b58773b"} Jul 10 02:23:13.306089 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.306068 2575 generic.go:356] "Generic (PLEG): container finished" podID="174a0477fbdfffd07715b9db16c916ce" containerID="1df89a36833739b1f4388446d3da81622a3bfc1b5ca0c7669eb94ecaff6ef58d" exitCode=0 Jul 10 02:23:13.306178 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.306131 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal" event={"ID":"174a0477fbdfffd07715b9db16c916ce","Type":"ContainerDied","Data":"1df89a36833739b1f4388446d3da81622a3bfc1b5ca0c7669eb94ecaff6ef58d"} Jul 10 02:23:13.307565 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.307539 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hv4z2" event={"ID":"62337cd8-e0f1-421f-9b88-e9a4f643841a","Type":"ContainerStarted","Data":"0d3a086be4dd3e62aa6662bcbaf42892165a15f21bae9813764712386b7b2cad"} Jul 10 02:23:13.308930 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.308910 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" event={"ID":"76708d83-5234-4c4a-b588-7fba2172d21a","Type":"ContainerStarted","Data":"ead32cf230b43edb29b7a259f2a157ae7d09373a92fc1fcfcd6a57f79e28b2e2"} Jul 10 02:23:13.310276 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.310236 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" event={"ID":"72680f75-f579-4701-a4a8-2ff305b2c047","Type":"ContainerStarted","Data":"5c96a3ce74f11276c1357422705728b13adb04c41981b95ed230612c3f44f021"} Jul 10 02:23:13.311726 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.311696 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-blpm7" event={"ID":"3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee","Type":"ContainerStarted","Data":"20d2d699dcc25c9399f62c7d2fb6ab3b3351823136aecbb54abc384b0b3003f5"} Jul 10 02:23:13.313071 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.313049 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mnprz" event={"ID":"33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b","Type":"ContainerStarted","Data":"42dec16bb86c177ae80a48c191544b1982ba0c1059bb426d7cb54cfa2f0a7812"} Jul 10 02:23:13.314624 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.314601 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d47h5" event={"ID":"a2929b6d-dfa7-448e-8560-235cddb333e8","Type":"ContainerStarted","Data":"22c4095fcb3e632ab8f2746230e4e83f1dc75f02745d1d6c0f6a88fd20b4aa57"} Jul 10 02:23:13.316882 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.316864 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:23:13.317298 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.317262 2575 generic.go:356] "Generic (PLEG): container finished" podID="61241733-b531-4151-a869-cc89000e2e99" containerID="67feaae39fb03ca3dc0c36efcd9f5aa81b28860ee0f811c3363a4e610af8a2cd" exitCode=1 Jul 10 02:23:13.317401 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.317278 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" event={"ID":"61241733-b531-4151-a869-cc89000e2e99","Type":"ContainerStarted","Data":"8704d8624beb30497439dde533eca690593f5096cc0c348aa6e9aa5cf67ff4b3"} Jul 10 02:23:13.317460 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.317420 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" event={"ID":"61241733-b531-4151-a869-cc89000e2e99","Type":"ContainerDied","Data":"67feaae39fb03ca3dc0c36efcd9f5aa81b28860ee0f811c3363a4e610af8a2cd"} Jul 10 02:23:13.317513 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.317479 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" event={"ID":"61241733-b531-4151-a869-cc89000e2e99","Type":"ContainerStarted","Data":"e1b9947ae7e9bf21ea70a4edf7795f4a0b0978ae11af360fa2c3ca1b8836d9c7"} Jul 10 02:23:13.318937 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.318913 2575 generic.go:356] "Generic (PLEG): container finished" podID="d74a1be9-254d-4ec7-a66a-e521e4d19dca" containerID="cd724e5f2ba5c172aba38d84608d9dc84601e5c0c5af929f6cac2e627621df83" exitCode=0 Jul 10 02:23:13.319017 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.318949 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd54c" event={"ID":"d74a1be9-254d-4ec7-a66a-e521e4d19dca","Type":"ContainerDied","Data":"cd724e5f2ba5c172aba38d84608d9dc84601e5c0c5af929f6cac2e627621df83"} Jul 10 02:23:13.335563 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.335523 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.ec2.internal" podStartSLOduration=22.335509853 podStartE2EDuration="22.335509853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:13.31949749 +0000 UTC m=+22.718950403" watchObservedRunningTime="2026-07-10 02:23:13.335509853 +0000 UTC m=+22.734962766" Jul 10 02:23:13.335988 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.335949 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gxxlp" podStartSLOduration=2.498346786 podStartE2EDuration="22.335938306s" totalImagesPullingTime="19.83759152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:13.33522306 +0000 UTC m=+22.734675974" watchObservedRunningTime="2026-07-10 02:23:13.335938306 +0000 UTC m=+22.735391285" Jul 10 02:23:13.350069 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.349896 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d47h5" podStartSLOduration=2.519297385 podStartE2EDuration="22.349882339s" totalImagesPullingTime="19.830584954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:13.349378942 +0000 UTC m=+22.748831856" watchObservedRunningTime="2026-07-10 02:23:13.349882339 +0000 UTC m=+22.749335251" Jul 10 02:23:13.363902 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.363853 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hv4z2" podStartSLOduration=2.536390638 podStartE2EDuration="22.36383512s" totalImagesPullingTime="19.827444482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:13.363216134 +0000 UTC m=+22.762669048" watchObservedRunningTime="2026-07-10 02:23:13.36383512 +0000 UTC m=+22.763288036" Jul 10 02:23:13.400599 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.400558 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mnprz" podStartSLOduration=2.501078024 podStartE2EDuration="22.400543533s" totalImagesPullingTime="19.899465509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:13.400040605 +0000 UTC m=+22.799493520" watchObservedRunningTime="2026-07-10 02:23:13.400543533 +0000 UTC m=+22.799996449" Jul 10 02:23:13.421500 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:13.421464 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-blpm7" podStartSLOduration=2.483837361 podStartE2EDuration="22.421451406s" totalImagesPullingTime="19.937614045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:13.421209521 +0000 UTC m=+22.820662434" watchObservedRunningTime="2026-07-10 02:23:13.421451406 +0000 UTC m=+22.820904344" Jul 10 02:23:14.123483 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:14.123454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:14.123644 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:14.123574 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:23:14.123941 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:14.123916 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:14.123941 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:14.123934 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:14.124079 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:14.124037 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:23:14.124143 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:14.124124 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:23:14.219397 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:14.219374 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 02:23:14.323472 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:14.323433 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal" event={"ID":"174a0477fbdfffd07715b9db16c916ce","Type":"ContainerStarted","Data":"8f829bcb8ee4d9f9c0748d0c6a38a7664174ea6eb86044e56b9c30c513100c8d"} Jul 10 02:23:14.325754 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:14.325730 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" event={"ID":"72680f75-f579-4701-a4a8-2ff305b2c047","Type":"ContainerStarted","Data":"be8bf65eacb5ac6993ea3881188bef7ff97665b154b3740d6a1690da686d25e2"} Jul 10 02:23:14.327294 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:14.327270 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mrv4l" event={"ID":"7f510e04-1bc2-4f2b-9ccd-3931b38ed667","Type":"ContainerStarted","Data":"a955c24cbeb000cf9a8678b0e8a3a288306c5d4e6feb88af2079a2fda20745bf"} Jul 10 02:23:14.330460 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:14.330434 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:23:14.330944 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:14.330913 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" event={"ID":"61241733-b531-4151-a869-cc89000e2e99","Type":"ContainerStarted","Data":"325a0e8ea3f9a0f17934cd3babe6741bb2ee8c353b3e3d437799ff4df878bfd3"} Jul 10 02:23:14.331030 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:14.330988 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" event={"ID":"61241733-b531-4151-a869-cc89000e2e99","Type":"ContainerStarted","Data":"052aa9e900f80c13a95fba9078f7fe068a48681d85efe00f55370ca5d9ce71fd"} Jul 10 02:23:14.331081 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:14.331041 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" event={"ID":"61241733-b531-4151-a869-cc89000e2e99","Type":"ContainerStarted","Data":"aacd5dfd343256199c127e5149321ebce076df19afe645e1b3e64eaa7106ec19"} Jul 10 02:23:14.343000 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:14.342963 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.ec2.internal" podStartSLOduration=23.342947127 podStartE2EDuration="23.342947127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:14.342454672 +0000 UTC m=+23.741907585" watchObservedRunningTime="2026-07-10 02:23:14.342947127 +0000 UTC m=+23.742400048" Jul 10 02:23:14.364024 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:14.363944 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mrv4l" podStartSLOduration=3.500496925 podStartE2EDuration="23.363936399s" totalImagesPullingTime="19.863439474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:14.363550363 +0000 UTC m=+23.763003277" watchObservedRunningTime="2026-07-10 02:23:14.363936399 +0000 UTC m=+23.763389312" Jul 10 02:23:15.076422 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:15.076311 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T02:23:14.21939456Z","UUID":"ee266309-0b50-47ba-892c-d10c80bab0e6","Handler":null,"Name":"","Endpoint":""} Jul 10 02:23:15.077717 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:15.077697 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 02:23:15.077828 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:15.077736 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 02:23:15.514339 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:15.514124 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mnprz" Jul 10 02:23:15.514744 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:15.514671 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mnprz" Jul 10 02:23:16.123828 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:16.123751 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:16.123828 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:16.123762 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:16.124039 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:16.123874 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:23:16.124039 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:16.123914 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:23:16.124039 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:16.123936 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:16.124039 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:16.124001 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:23:16.338385 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:16.338359 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:23:16.338778 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:16.338749 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" event={"ID":"61241733-b531-4151-a869-cc89000e2e99","Type":"ContainerStarted","Data":"e88733d60f6bdef334014f2efe9f36e54e12d638d04158d9af24f7a2b8958351"} Jul 10 02:23:16.340634 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:16.340609 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" event={"ID":"72680f75-f579-4701-a4a8-2ff305b2c047","Type":"ContainerStarted","Data":"171633cf5fa2736146310221ba96cc9075def4056b6ce094173da5c2ca57ce4b"} Jul 10 02:23:16.360664 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:16.360619 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tbtt" podStartSLOduration=3.302860861 podStartE2EDuration="25.360606409s" totalImagesPullingTime="22.057745548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:22:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:16.360229181 +0000 UTC m=+25.759682094" watchObservedRunningTime="2026-07-10 02:23:16.360606409 +0000 UTC m=+25.760059321" Jul 10 02:23:18.123723 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:18.123639 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:18.123723 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:18.123711 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:18.124338 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:18.123813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:18.124338 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:18.123820 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:23:18.124338 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:18.123894 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:23:18.124338 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:18.123985 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:23:19.348799 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:19.348538 2575 generic.go:356] "Generic (PLEG): container finished" podID="d74a1be9-254d-4ec7-a66a-e521e4d19dca" containerID="4f60c61b09f0cdf8418ce88720b919180b499b2283a9e5fd3a64848bafa36fa4" exitCode=0 Jul 10 02:23:19.348799 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:19.348621 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd54c" event={"ID":"d74a1be9-254d-4ec7-a66a-e521e4d19dca","Type":"ContainerDied","Data":"4f60c61b09f0cdf8418ce88720b919180b499b2283a9e5fd3a64848bafa36fa4"} Jul 10 02:23:19.352349 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:19.352330 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:23:19.352717 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:19.352692 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" event={"ID":"61241733-b531-4151-a869-cc89000e2e99","Type":"ContainerStarted","Data":"be0245f31ffd0ff60d7161089674377fb058177cd34ab8936ef2ccbf7a92ec90"} Jul 10 02:23:19.352990 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:19.352965 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:23:19.353102 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:19.353024 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:23:19.353195 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:19.353167 2575 scope.go:117] "RemoveContainer" containerID="67feaae39fb03ca3dc0c36efcd9f5aa81b28860ee0f811c3363a4e610af8a2cd" Jul 10 02:23:19.369753 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:19.369730 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:23:19.888343 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:19.888315 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mnprz" Jul 10 02:23:19.888545 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:19.888436 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 02:23:19.888905 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:19.888883 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mnprz" Jul 10 02:23:20.123724 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:20.123661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:20.123724 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:20.123697 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:20.123724 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:20.123697 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:20.124203 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:20.123809 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:23:20.124203 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:20.123908 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:23:20.124203 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:20.123998 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:23:20.357806 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:20.357741 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:23:20.358178 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:20.358009 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" event={"ID":"61241733-b531-4151-a869-cc89000e2e99","Type":"ContainerStarted","Data":"328cbb46b250d7919e2abc5fc0c48df8f32d9e5176a14d537eb50cc7a37baccc"} Jul 10 02:23:20.358283 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:20.358267 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:23:20.375696 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:20.375674 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:23:20.405063 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:20.404100 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" podStartSLOduration=8.87367147 podStartE2EDuration="29.404082716s" totalImagesPullingTime="20.530411246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:20.403936363 +0000 UTC m=+29.803389278" watchObservedRunningTime="2026-07-10 02:23:20.404082716 +0000 UTC m=+29.803535630" Jul 10 02:23:20.822375 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:20.822145 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lgw26"] Jul 10 02:23:20.822509 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:20.822462 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:20.822563 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:20.822548 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:23:20.825371 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:20.825348 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j28gk"] Jul 10 02:23:20.825499 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:20.825448 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:20.825558 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:20.825541 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:23:20.825959 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:20.825942 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r4t9z"] Jul 10 02:23:20.826045 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:20.826032 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:20.826154 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:20.826119 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:23:21.362035 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:21.361965 2575 generic.go:356] "Generic (PLEG): container finished" podID="d74a1be9-254d-4ec7-a66a-e521e4d19dca" containerID="0e02a612c02c2190ef4bc23ac68dca46676bd4b916b505c4cdb6c2f827641c0d" exitCode=0 Jul 10 02:23:21.362492 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:21.362056 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd54c" event={"ID":"d74a1be9-254d-4ec7-a66a-e521e4d19dca","Type":"ContainerDied","Data":"0e02a612c02c2190ef4bc23ac68dca46676bd4b916b505c4cdb6c2f827641c0d"} Jul 10 02:23:22.123496 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:22.123426 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:22.123633 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:22.123541 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:23:22.123633 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:22.123604 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:22.123743 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:22.123724 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:23:23.124166 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:23.124134 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:23.124747 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:23.124232 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:23:23.368198 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:23.368113 2575 generic.go:356] "Generic (PLEG): container finished" podID="d74a1be9-254d-4ec7-a66a-e521e4d19dca" containerID="13b1443c26d0994f865a3b725b786193bb0e30368ea6ec212347afc83718fa4c" exitCode=0 Jul 10 02:23:23.368198 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:23.368163 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd54c" event={"ID":"d74a1be9-254d-4ec7-a66a-e521e4d19dca","Type":"ContainerDied","Data":"13b1443c26d0994f865a3b725b786193bb0e30368ea6ec212347afc83718fa4c"} Jul 10 02:23:23.767473 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:23.767391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:23.767609 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:23.767493 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:23:23.767609 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:23.767539 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret podName:82138b66-c754-4c2d-a828-a104a92a9968 nodeName:}" failed. No retries permitted until 2026-07-10 02:23:55.767526906 +0000 UTC m=+65.166979800 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret") pod "global-pull-secret-syncer-lgw26" (UID: "82138b66-c754-4c2d-a828-a104a92a9968") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:23:23.868259 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:23.868217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs\") pod \"network-metrics-daemon-r4t9z\" (UID: \"90c11f6c-5762-4634-9dad-5dc5e5b751eb\") " pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:23.868409 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:23.868330 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:23:23.868409 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:23.868380 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs podName:90c11f6c-5762-4634-9dad-5dc5e5b751eb nodeName:}" failed. No retries permitted until 2026-07-10 02:23:55.868365956 +0000 UTC m=+65.267818847 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs") pod "network-metrics-daemon-r4t9z" (UID: "90c11f6c-5762-4634-9dad-5dc5e5b751eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:23:23.968621 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:23.968585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnt5v\" (UniqueName: \"kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v\") pod \"network-check-target-j28gk\" (UID: \"e5e656b6-b264-455f-9da1-2de1fbac1a9f\") " pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:23.968796 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:23.968767 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:23:23.968796 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:23.968790 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:23:23.968911 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:23.968804 2575 projected.go:196] Error preparing data for projected volume kube-api-access-tnt5v for pod openshift-network-diagnostics/network-check-target-j28gk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:23:23.968911 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:23.968866 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v podName:e5e656b6-b264-455f-9da1-2de1fbac1a9f nodeName:}" failed. No retries permitted until 2026-07-10 02:23:55.968846368 +0000 UTC m=+65.368299271 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnt5v" (UniqueName: "kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v") pod "network-check-target-j28gk" (UID: "e5e656b6-b264-455f-9da1-2de1fbac1a9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:23:24.123491 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:24.123413 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:24.123491 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:24.123412 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:24.123700 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:24.123549 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgw26" podUID="82138b66-c754-4c2d-a828-a104a92a9968" Jul 10 02:23:24.123700 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:24.123645 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4t9z" podUID="90c11f6c-5762-4634-9dad-5dc5e5b751eb" Jul 10 02:23:25.123708 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.123672 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:25.124164 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:25.123809 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j28gk" podUID="e5e656b6-b264-455f-9da1-2de1fbac1a9f" Jul 10 02:23:25.358554 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.358351 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-69.ec2.internal" event="NodeReady" Jul 10 02:23:25.358709 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.358674 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 02:23:25.392930 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.392848 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6459bf6464-xcsdp"] Jul 10 02:23:25.394831 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.394805 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.397800 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.397778 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 02:23:25.397924 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.397822 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zf7mn\"" Jul 10 02:23:25.397924 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.397891 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 02:23:25.397924 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.397917 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 02:23:25.403108 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.403088 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 02:23:25.409304 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.408939 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5drj5"] Jul 10 02:23:25.411306 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.411288 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6459bf6464-xcsdp"] Jul 10 02:23:25.411426 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.411410 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:25.413928 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.413907 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 02:23:25.414022 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.413989 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qdhtc\"" Jul 10 02:23:25.414105 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.414089 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 02:23:25.414852 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.414729 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b79d4"] Jul 10 02:23:25.416593 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.416573 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b79d4" Jul 10 02:23:25.419198 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.418956 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c8vpj\"" Jul 10 02:23:25.419198 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.418958 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 02:23:25.419198 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.419078 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 02:23:25.419198 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.418959 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 02:23:25.427049 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.426993 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5drj5"] Jul 10 02:23:25.427458 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.427434 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b79d4"] Jul 10 02:23:25.478659 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.478632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:25.478659 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.478663 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec1f23d1-33ad-495d-b737-ea19ddc98b69-cert\") pod \"ingress-canary-b79d4\" (UID: \"ec1f23d1-33ad-495d-b737-ea19ddc98b69\") " pod="openshift-ingress-canary/ingress-canary-b79d4" Jul 10 02:23:25.478866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.478680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrxf2\" (UniqueName: \"kubernetes.io/projected/043938f8-88f9-438f-a386-a0bf37ef9b1f-kube-api-access-wrxf2\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:25.478866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.478696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29wsz\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-kube-api-access-29wsz\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.478866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.478716 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/076c71ed-cbf4-4324-be05-ca3f80c1db93-image-registry-private-configuration\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.478866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.478741 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-certificates\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.478866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.478765 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043938f8-88f9-438f-a386-a0bf37ef9b1f-tmp-dir\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:25.478866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.478782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-bound-sa-token\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.478866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.478802 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.478866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.478818 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/076c71ed-cbf4-4324-be05-ca3f80c1db93-trusted-ca\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.479135 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.478885 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/043938f8-88f9-438f-a386-a0bf37ef9b1f-config-volume\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:25.479135 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.478908 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpqdk\" (UniqueName: \"kubernetes.io/projected/ec1f23d1-33ad-495d-b737-ea19ddc98b69-kube-api-access-vpqdk\") pod \"ingress-canary-b79d4\" (UID: \"ec1f23d1-33ad-495d-b737-ea19ddc98b69\") " pod="openshift-ingress-canary/ingress-canary-b79d4" Jul 10 02:23:25.479135 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.478936 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/076c71ed-cbf4-4324-be05-ca3f80c1db93-ca-trust-extracted\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.479135 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.478965 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/076c71ed-cbf4-4324-be05-ca3f80c1db93-installation-pull-secrets\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.579783 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.579756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec1f23d1-33ad-495d-b737-ea19ddc98b69-cert\") pod \"ingress-canary-b79d4\" (UID: \"ec1f23d1-33ad-495d-b737-ea19ddc98b69\") " pod="openshift-ingress-canary/ingress-canary-b79d4" Jul 10 02:23:25.579965 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.579794 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrxf2\" (UniqueName: \"kubernetes.io/projected/043938f8-88f9-438f-a386-a0bf37ef9b1f-kube-api-access-wrxf2\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:25.579965 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.579818 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29wsz\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-kube-api-access-29wsz\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.580077 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.579959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/076c71ed-cbf4-4324-be05-ca3f80c1db93-image-registry-private-configuration\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.580077 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.580006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-certificates\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.580077 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.580037 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043938f8-88f9-438f-a386-a0bf37ef9b1f-tmp-dir\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:25.580077 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.580061 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-bound-sa-token\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.580294 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.580097 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.580294 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.580118 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/076c71ed-cbf4-4324-be05-ca3f80c1db93-trusted-ca\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.580294 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.580164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/043938f8-88f9-438f-a386-a0bf37ef9b1f-config-volume\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:25.580294 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.580191 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpqdk\" (UniqueName: \"kubernetes.io/projected/ec1f23d1-33ad-495d-b737-ea19ddc98b69-kube-api-access-vpqdk\") pod \"ingress-canary-b79d4\" (UID: \"ec1f23d1-33ad-495d-b737-ea19ddc98b69\") " pod="openshift-ingress-canary/ingress-canary-b79d4" Jul 10 02:23:25.580294 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.580230 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/076c71ed-cbf4-4324-be05-ca3f80c1db93-ca-trust-extracted\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.580294 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:25.580266 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 02:23:25.580294 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:25.580283 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6459bf6464-xcsdp: secret "image-registry-tls" not found Jul 10 02:23:25.580612 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.580299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/076c71ed-cbf4-4324-be05-ca3f80c1db93-installation-pull-secrets\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.580612 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.580337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:25.580612 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:25.580353 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls podName:076c71ed-cbf4-4324-be05-ca3f80c1db93 nodeName:}" failed. No retries permitted until 2026-07-10 02:23:26.08033214 +0000 UTC m=+35.479785046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls") pod "image-registry-6459bf6464-xcsdp" (UID: "076c71ed-cbf4-4324-be05-ca3f80c1db93") : secret "image-registry-tls" not found Jul 10 02:23:25.580612 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.580378 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043938f8-88f9-438f-a386-a0bf37ef9b1f-tmp-dir\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:25.580612 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:25.580401 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:23:25.580612 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:25.580453 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls podName:043938f8-88f9-438f-a386-a0bf37ef9b1f nodeName:}" failed. No retries permitted until 2026-07-10 02:23:26.080439478 +0000 UTC m=+35.479892387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls") pod "dns-default-5drj5" (UID: "043938f8-88f9-438f-a386-a0bf37ef9b1f") : secret "dns-default-metrics-tls" not found Jul 10 02:23:25.580901 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.580655 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-certificates\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.580901 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.580800 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/076c71ed-cbf4-4324-be05-ca3f80c1db93-ca-trust-extracted\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.581143 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.581098 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/043938f8-88f9-438f-a386-a0bf37ef9b1f-config-volume\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:25.581239 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.581189 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/076c71ed-cbf4-4324-be05-ca3f80c1db93-trusted-ca\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.584709 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.584682 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/076c71ed-cbf4-4324-be05-ca3f80c1db93-installation-pull-secrets\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.584860 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.584835 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/076c71ed-cbf4-4324-be05-ca3f80c1db93-image-registry-private-configuration\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.584980 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.584885 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec1f23d1-33ad-495d-b737-ea19ddc98b69-cert\") pod \"ingress-canary-b79d4\" (UID: \"ec1f23d1-33ad-495d-b737-ea19ddc98b69\") " pod="openshift-ingress-canary/ingress-canary-b79d4" Jul 10 02:23:25.591403 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.591365 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrxf2\" (UniqueName: \"kubernetes.io/projected/043938f8-88f9-438f-a386-a0bf37ef9b1f-kube-api-access-wrxf2\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:25.591657 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.591634 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpqdk\" (UniqueName: \"kubernetes.io/projected/ec1f23d1-33ad-495d-b737-ea19ddc98b69-kube-api-access-vpqdk\") pod \"ingress-canary-b79d4\" (UID: \"ec1f23d1-33ad-495d-b737-ea19ddc98b69\") " pod="openshift-ingress-canary/ingress-canary-b79d4" Jul 10 02:23:25.591853 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.591834 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-bound-sa-token\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.592018 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.591997 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29wsz\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-kube-api-access-29wsz\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:25.730149 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.730047 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b79d4" Jul 10 02:23:25.867940 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:25.867901 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b79d4"] Jul 10 02:23:25.871591 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:23:25.871562 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1f23d1_33ad_495d_b737_ea19ddc98b69.slice/crio-96cdd1443db2445501cb732078d0180174c1d0532205bb59edf8214a9660f88a WatchSource:0}: Error finding container 96cdd1443db2445501cb732078d0180174c1d0532205bb59edf8214a9660f88a: Status 404 returned error can't find the container with id 96cdd1443db2445501cb732078d0180174c1d0532205bb59edf8214a9660f88a Jul 10 02:23:26.083602 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:26.083528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:26.083602 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:26.083570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:26.083758 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:26.083653 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 02:23:26.083758 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:26.083663 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6459bf6464-xcsdp: secret "image-registry-tls" not found Jul 10 02:23:26.083758 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:26.083666 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:23:26.083758 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:26.083715 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls podName:076c71ed-cbf4-4324-be05-ca3f80c1db93 nodeName:}" failed. No retries permitted until 2026-07-10 02:23:27.083702822 +0000 UTC m=+36.483155713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls") pod "image-registry-6459bf6464-xcsdp" (UID: "076c71ed-cbf4-4324-be05-ca3f80c1db93") : secret "image-registry-tls" not found Jul 10 02:23:26.083758 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:26.083728 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls podName:043938f8-88f9-438f-a386-a0bf37ef9b1f nodeName:}" failed. No retries permitted until 2026-07-10 02:23:27.08372284 +0000 UTC m=+36.483175730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls") pod "dns-default-5drj5" (UID: "043938f8-88f9-438f-a386-a0bf37ef9b1f") : secret "dns-default-metrics-tls" not found Jul 10 02:23:26.123834 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:26.123806 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:26.124269 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:26.123965 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:26.126638 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:26.126620 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gcczx\"" Jul 10 02:23:26.126763 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:26.126666 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 02:23:26.127751 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:26.127732 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 02:23:26.376548 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:26.376469 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b79d4" event={"ID":"ec1f23d1-33ad-495d-b737-ea19ddc98b69","Type":"ContainerStarted","Data":"96cdd1443db2445501cb732078d0180174c1d0532205bb59edf8214a9660f88a"} Jul 10 02:23:27.091584 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:27.091541 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:27.091734 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:27.091623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:27.091734 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:27.091711 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:23:27.091838 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:27.091746 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 02:23:27.091838 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:27.091760 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6459bf6464-xcsdp: secret "image-registry-tls" not found Jul 10 02:23:27.091838 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:27.091785 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls podName:043938f8-88f9-438f-a386-a0bf37ef9b1f nodeName:}" failed. No retries permitted until 2026-07-10 02:23:29.091765134 +0000 UTC m=+38.491218046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls") pod "dns-default-5drj5" (UID: "043938f8-88f9-438f-a386-a0bf37ef9b1f") : secret "dns-default-metrics-tls" not found Jul 10 02:23:27.091838 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:27.091803 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls podName:076c71ed-cbf4-4324-be05-ca3f80c1db93 nodeName:}" failed. No retries permitted until 2026-07-10 02:23:29.091794625 +0000 UTC m=+38.491247517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls") pod "image-registry-6459bf6464-xcsdp" (UID: "076c71ed-cbf4-4324-be05-ca3f80c1db93") : secret "image-registry-tls" not found Jul 10 02:23:27.123792 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:27.123763 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:27.126654 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:27.126627 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4r7lb\"" Jul 10 02:23:27.127612 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:27.127589 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 02:23:27.127612 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:27.127604 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 02:23:28.300006 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.299974 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-fkdsw"] Jul 10 02:23:28.302160 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.301996 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr"] Jul 10 02:23:28.302397 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.302368 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.303898 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.303879 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" Jul 10 02:23:28.306925 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.306905 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 10 02:23:28.307259 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.307210 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 02:23:28.307362 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.307214 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:23:28.308444 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.308271 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 10 02:23:28.308444 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.308289 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 10 02:23:28.308444 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.308339 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gmrqs\"" Jul 10 02:23:28.308444 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.308441 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xxm5q\"" Jul 10 02:23:28.308678 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.308465 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 02:23:28.308678 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.308475 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 10 02:23:28.312991 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.312967 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 10 02:23:28.319667 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.319202 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-fkdsw"] Jul 10 02:23:28.320080 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.320057 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr"] Jul 10 02:23:28.402351 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.402320 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ce04276-ac1b-463f-b812-5dd0b2c82145-tmp\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.402516 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.402374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ce04276-ac1b-463f-b812-5dd0b2c82145-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.402516 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.402425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ce04276-ac1b-463f-b812-5dd0b2c82145-serving-cert\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.402641 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.402520 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ce04276-ac1b-463f-b812-5dd0b2c82145-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.402641 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.402556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkslg\" (UniqueName: \"kubernetes.io/projected/7955372c-7076-4a1a-aff4-6cf9898d2576-kube-api-access-vkslg\") pod \"cluster-samples-operator-65db858894-5b5xr\" (UID: \"7955372c-7076-4a1a-aff4-6cf9898d2576\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" Jul 10 02:23:28.402641 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.402587 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hct6j\" (UniqueName: \"kubernetes.io/projected/0ce04276-ac1b-463f-b812-5dd0b2c82145-kube-api-access-hct6j\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.402641 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.402620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5b5xr\" (UID: \"7955372c-7076-4a1a-aff4-6cf9898d2576\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" Jul 10 02:23:28.402839 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.402672 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0ce04276-ac1b-463f-b812-5dd0b2c82145-snapshots\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.503736 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.503705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ce04276-ac1b-463f-b812-5dd0b2c82145-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.503910 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.503753 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkslg\" (UniqueName: \"kubernetes.io/projected/7955372c-7076-4a1a-aff4-6cf9898d2576-kube-api-access-vkslg\") pod \"cluster-samples-operator-65db858894-5b5xr\" (UID: \"7955372c-7076-4a1a-aff4-6cf9898d2576\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" Jul 10 02:23:28.503910 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.503786 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hct6j\" (UniqueName: \"kubernetes.io/projected/0ce04276-ac1b-463f-b812-5dd0b2c82145-kube-api-access-hct6j\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.503910 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.503816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5b5xr\" (UID: \"7955372c-7076-4a1a-aff4-6cf9898d2576\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" Jul 10 02:23:28.503910 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.503845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0ce04276-ac1b-463f-b812-5dd0b2c82145-snapshots\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.504129 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:28.503947 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:23:28.504129 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:28.504024 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls podName:7955372c-7076-4a1a-aff4-6cf9898d2576 nodeName:}" failed. No retries permitted until 2026-07-10 02:23:29.004009401 +0000 UTC m=+38.403462307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls") pod "cluster-samples-operator-65db858894-5b5xr" (UID: "7955372c-7076-4a1a-aff4-6cf9898d2576") : secret "samples-operator-tls" not found Jul 10 02:23:28.504129 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.503950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ce04276-ac1b-463f-b812-5dd0b2c82145-tmp\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.504129 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.504125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ce04276-ac1b-463f-b812-5dd0b2c82145-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.504364 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.504153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ce04276-ac1b-463f-b812-5dd0b2c82145-serving-cert\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.504419 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.504373 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ce04276-ac1b-463f-b812-5dd0b2c82145-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.504419 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.504384 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ce04276-ac1b-463f-b812-5dd0b2c82145-tmp\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.504514 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.504482 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0ce04276-ac1b-463f-b812-5dd0b2c82145-snapshots\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.505007 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.504983 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ce04276-ac1b-463f-b812-5dd0b2c82145-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.507490 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.507469 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ce04276-ac1b-463f-b812-5dd0b2c82145-serving-cert\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.519130 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.519107 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hct6j\" (UniqueName: \"kubernetes.io/projected/0ce04276-ac1b-463f-b812-5dd0b2c82145-kube-api-access-hct6j\") pod \"insights-operator-68d9dd94cf-fkdsw\" (UID: \"0ce04276-ac1b-463f-b812-5dd0b2c82145\") " pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:28.519576 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.519555 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkslg\" (UniqueName: \"kubernetes.io/projected/7955372c-7076-4a1a-aff4-6cf9898d2576-kube-api-access-vkslg\") pod \"cluster-samples-operator-65db858894-5b5xr\" (UID: \"7955372c-7076-4a1a-aff4-6cf9898d2576\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" Jul 10 02:23:28.615399 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:28.615324 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" Jul 10 02:23:29.008070 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:29.007986 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5b5xr\" (UID: \"7955372c-7076-4a1a-aff4-6cf9898d2576\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" Jul 10 02:23:29.008207 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:29.008149 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:23:29.008264 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:29.008221 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls podName:7955372c-7076-4a1a-aff4-6cf9898d2576 nodeName:}" failed. No retries permitted until 2026-07-10 02:23:30.008203095 +0000 UTC m=+39.407655986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls") pod "cluster-samples-operator-65db858894-5b5xr" (UID: "7955372c-7076-4a1a-aff4-6cf9898d2576") : secret "samples-operator-tls" not found Jul 10 02:23:29.108402 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:29.108364 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:29.108587 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:29.108447 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:29.108587 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:29.108553 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:23:29.108587 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:29.108567 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 02:23:29.108587 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:29.108589 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6459bf6464-xcsdp: secret "image-registry-tls" not found Jul 10 02:23:29.108786 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:29.108632 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls podName:043938f8-88f9-438f-a386-a0bf37ef9b1f nodeName:}" failed. No retries permitted until 2026-07-10 02:23:33.108612244 +0000 UTC m=+42.508065152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls") pod "dns-default-5drj5" (UID: "043938f8-88f9-438f-a386-a0bf37ef9b1f") : secret "dns-default-metrics-tls" not found Jul 10 02:23:29.108786 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:29.108653 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls podName:076c71ed-cbf4-4324-be05-ca3f80c1db93 nodeName:}" failed. No retries permitted until 2026-07-10 02:23:33.108643406 +0000 UTC m=+42.508096303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls") pod "image-registry-6459bf6464-xcsdp" (UID: "076c71ed-cbf4-4324-be05-ca3f80c1db93") : secret "image-registry-tls" not found Jul 10 02:23:30.015676 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:30.015493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5b5xr\" (UID: \"7955372c-7076-4a1a-aff4-6cf9898d2576\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" Jul 10 02:23:30.015676 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:30.015666 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:23:30.016184 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:30.015733 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls podName:7955372c-7076-4a1a-aff4-6cf9898d2576 nodeName:}" failed. No retries permitted until 2026-07-10 02:23:32.015716921 +0000 UTC m=+41.415169812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls") pod "cluster-samples-operator-65db858894-5b5xr" (UID: "7955372c-7076-4a1a-aff4-6cf9898d2576") : secret "samples-operator-tls" not found Jul 10 02:23:30.500334 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:30.500303 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-fkdsw"] Jul 10 02:23:30.504166 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:23:30.504144 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce04276_ac1b_463f_b812_5dd0b2c82145.slice/crio-dd47f94ac19c20e0ddce28497ba09490ed478960f61f2ca7f96376be22b236b8 WatchSource:0}: Error finding container dd47f94ac19c20e0ddce28497ba09490ed478960f61f2ca7f96376be22b236b8: Status 404 returned error can't find the container with id dd47f94ac19c20e0ddce28497ba09490ed478960f61f2ca7f96376be22b236b8 Jul 10 02:23:31.391925 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:31.391694 2575 generic.go:356] "Generic (PLEG): container finished" podID="d74a1be9-254d-4ec7-a66a-e521e4d19dca" containerID="d160e9dec0b61d01dc315d36a102f824c6892bd6b54d539e2835c985f914e8fd" exitCode=0 Jul 10 02:23:31.392710 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:31.391780 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd54c" event={"ID":"d74a1be9-254d-4ec7-a66a-e521e4d19dca","Type":"ContainerDied","Data":"d160e9dec0b61d01dc315d36a102f824c6892bd6b54d539e2835c985f914e8fd"} Jul 10 02:23:31.393697 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:31.393674 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b79d4" event={"ID":"ec1f23d1-33ad-495d-b737-ea19ddc98b69","Type":"ContainerStarted","Data":"eab46d589b480221542c81ed1da6989e1d447d05dea782a673ed351c6e2c6969"} Jul 10 02:23:31.394844 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:31.394820 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" event={"ID":"0ce04276-ac1b-463f-b812-5dd0b2c82145","Type":"ContainerStarted","Data":"dd47f94ac19c20e0ddce28497ba09490ed478960f61f2ca7f96376be22b236b8"} Jul 10 02:23:31.436931 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:31.436878 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b79d4" podStartSLOduration=1.9396488 podStartE2EDuration="6.436859293s" totalImagesPullingTime="4.497210493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:31.436104405 +0000 UTC m=+40.835557319" watchObservedRunningTime="2026-07-10 02:23:31.436859293 +0000 UTC m=+40.836312208" Jul 10 02:23:32.032897 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:32.032853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5b5xr\" (UID: \"7955372c-7076-4a1a-aff4-6cf9898d2576\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" Jul 10 02:23:32.033095 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:32.032975 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:23:32.033095 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:32.033060 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls podName:7955372c-7076-4a1a-aff4-6cf9898d2576 nodeName:}" failed. No retries permitted until 2026-07-10 02:23:36.033037041 +0000 UTC m=+45.432489951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls") pod "cluster-samples-operator-65db858894-5b5xr" (UID: "7955372c-7076-4a1a-aff4-6cf9898d2576") : secret "samples-operator-tls" not found Jul 10 02:23:32.401309 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:32.401211 2575 generic.go:356] "Generic (PLEG): container finished" podID="d74a1be9-254d-4ec7-a66a-e521e4d19dca" containerID="8008273dc24c6288ddb0c84052b5a0c2a32a6363e4b1ac5d256402f6ac993263" exitCode=0 Jul 10 02:23:32.401309 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:32.401282 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd54c" event={"ID":"d74a1be9-254d-4ec7-a66a-e521e4d19dca","Type":"ContainerDied","Data":"8008273dc24c6288ddb0c84052b5a0c2a32a6363e4b1ac5d256402f6ac993263"} Jul 10 02:23:33.143799 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:33.143723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:33.143799 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:33.143779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:33.143967 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:33.143871 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:23:33.143967 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:33.143925 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls podName:043938f8-88f9-438f-a386-a0bf37ef9b1f nodeName:}" failed. No retries permitted until 2026-07-10 02:23:41.143911753 +0000 UTC m=+50.543364648 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls") pod "dns-default-5drj5" (UID: "043938f8-88f9-438f-a386-a0bf37ef9b1f") : secret "dns-default-metrics-tls" not found Jul 10 02:23:33.143967 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:33.143874 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 02:23:33.144060 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:33.143977 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6459bf6464-xcsdp: secret "image-registry-tls" not found Jul 10 02:23:33.144060 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:33.144025 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls podName:076c71ed-cbf4-4324-be05-ca3f80c1db93 nodeName:}" failed. No retries permitted until 2026-07-10 02:23:41.144013397 +0000 UTC m=+50.543466289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls") pod "image-registry-6459bf6464-xcsdp" (UID: "076c71ed-cbf4-4324-be05-ca3f80c1db93") : secret "image-registry-tls" not found Jul 10 02:23:33.407134 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:33.407042 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd54c" event={"ID":"d74a1be9-254d-4ec7-a66a-e521e4d19dca","Type":"ContainerStarted","Data":"9094af54cfbccaf9985d9be93ecc9e0675e0cc8c9939087b9ae5371a036980d3"} Jul 10 02:23:33.408542 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:33.408515 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" event={"ID":"0ce04276-ac1b-463f-b812-5dd0b2c82145","Type":"ContainerStarted","Data":"5c0e9d75bbe91bd78f7b360fa6193eb8430ecabbe27278e861b5f393a397e5bd"} Jul 10 02:23:33.432451 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:33.432408 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sd54c" podStartSLOduration=7.542883418 podStartE2EDuration="42.43239322s" totalImagesPullingTime="34.889509802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:22:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:33.430622757 +0000 UTC m=+42.830075673" watchObservedRunningTime="2026-07-10 02:23:33.43239322 +0000 UTC m=+42.831846133" Jul 10 02:23:33.448968 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:33.448929 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" podStartSLOduration=3.163975115 podStartE2EDuration="5.44891529s" totalImagesPullingTime="2.284940175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:33.448236494 +0000 UTC m=+42.847689407" watchObservedRunningTime="2026-07-10 02:23:33.44891529 +0000 UTC m=+42.848368247" Jul 10 02:23:34.402317 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.402277 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qzjnp"] Jul 10 02:23:34.421989 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.421964 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:34.425258 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.425217 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 02:23:34.425393 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.425217 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 02:23:34.426187 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.426168 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pfnlm\"" Jul 10 02:23:34.429772 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.429752 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qzjnp"] Jul 10 02:23:34.453106 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.453074 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/35865d4b-e324-4102-b2c6-0cfee7b6c10d-data-volume\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:34.453365 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.453328 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7rv8\" (UniqueName: \"kubernetes.io/projected/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-api-access-g7rv8\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:34.453487 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.453425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/35865d4b-e324-4102-b2c6-0cfee7b6c10d-crio-socket\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:34.453612 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.453574 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:34.453707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.453635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:34.554666 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.554634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:34.554666 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.554668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:34.554842 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.554720 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/35865d4b-e324-4102-b2c6-0cfee7b6c10d-data-volume\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:34.554842 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.554749 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7rv8\" (UniqueName: \"kubernetes.io/projected/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-api-access-g7rv8\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:34.554842 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:34.554771 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:23:34.554842 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.554773 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/35865d4b-e324-4102-b2c6-0cfee7b6c10d-crio-socket\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:34.554842 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:34.554830 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 02:23:34.555061 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:34.554837 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls podName:35865d4b-e324-4102-b2c6-0cfee7b6c10d nodeName:}" failed. No retries permitted until 2026-07-10 02:23:35.054818099 +0000 UTC m=+44.454271012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qzjnp" (UID: "35865d4b-e324-4102-b2c6-0cfee7b6c10d") : secret "insights-runtime-extractor-tls" not found Jul 10 02:23:34.555061 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:34.554912 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm podName:35865d4b-e324-4102-b2c6-0cfee7b6c10d nodeName:}" failed. No retries permitted until 2026-07-10 02:23:35.054894053 +0000 UTC m=+44.454346946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qzjnp" (UID: "35865d4b-e324-4102-b2c6-0cfee7b6c10d") : configmap "kube-rbac-proxy" not found Jul 10 02:23:34.555061 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.554937 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/35865d4b-e324-4102-b2c6-0cfee7b6c10d-crio-socket\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:34.555239 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.555098 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/35865d4b-e324-4102-b2c6-0cfee7b6c10d-data-volume\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:34.567629 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:34.567608 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7rv8\" (UniqueName: \"kubernetes.io/projected/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-api-access-g7rv8\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:35.059447 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:35.059413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:35.059447 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:35.059447 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:35.059663 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:35.059519 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 02:23:35.059663 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:35.059562 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm podName:35865d4b-e324-4102-b2c6-0cfee7b6c10d nodeName:}" failed. No retries permitted until 2026-07-10 02:23:36.059550805 +0000 UTC m=+45.459003695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qzjnp" (UID: "35865d4b-e324-4102-b2c6-0cfee7b6c10d") : configmap "kube-rbac-proxy" not found Jul 10 02:23:35.059663 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:35.059572 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:23:35.059663 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:35.059626 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls podName:35865d4b-e324-4102-b2c6-0cfee7b6c10d nodeName:}" failed. No retries permitted until 2026-07-10 02:23:36.059609432 +0000 UTC m=+45.459062323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qzjnp" (UID: "35865d4b-e324-4102-b2c6-0cfee7b6c10d") : secret "insights-runtime-extractor-tls" not found Jul 10 02:23:36.067025 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:36.066983 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:36.067025 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:36.067024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:36.067500 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:36.067053 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5b5xr\" (UID: \"7955372c-7076-4a1a-aff4-6cf9898d2576\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" Jul 10 02:23:36.067500 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:36.067114 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 02:23:36.067500 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:36.067132 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:23:36.067500 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:36.067145 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:23:36.067500 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:36.067185 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm podName:35865d4b-e324-4102-b2c6-0cfee7b6c10d nodeName:}" failed. No retries permitted until 2026-07-10 02:23:38.067166127 +0000 UTC m=+47.466619018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qzjnp" (UID: "35865d4b-e324-4102-b2c6-0cfee7b6c10d") : configmap "kube-rbac-proxy" not found Jul 10 02:23:36.067500 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:36.067199 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls podName:35865d4b-e324-4102-b2c6-0cfee7b6c10d nodeName:}" failed. No retries permitted until 2026-07-10 02:23:38.067192791 +0000 UTC m=+47.466645683 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qzjnp" (UID: "35865d4b-e324-4102-b2c6-0cfee7b6c10d") : secret "insights-runtime-extractor-tls" not found Jul 10 02:23:36.067500 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:36.067211 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls podName:7955372c-7076-4a1a-aff4-6cf9898d2576 nodeName:}" failed. No retries permitted until 2026-07-10 02:23:44.067204309 +0000 UTC m=+53.466657200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls") pod "cluster-samples-operator-65db858894-5b5xr" (UID: "7955372c-7076-4a1a-aff4-6cf9898d2576") : secret "samples-operator-tls" not found Jul 10 02:23:36.372725 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:36.372701 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d47h5_a2929b6d-dfa7-448e-8560-235cddb333e8/dns-node-resolver/0.log" Jul 10 02:23:37.367400 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:37.367374 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hv4z2_62337cd8-e0f1-421f-9b88-e9a4f643841a/node-ca/0.log" Jul 10 02:23:37.985457 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:37.985425 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-qcx9w"] Jul 10 02:23:38.019743 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.019716 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-qcx9w"] Jul 10 02:23:38.019876 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.019813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-qcx9w" Jul 10 02:23:38.026106 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.025819 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 10 02:23:38.026535 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.026519 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 10 02:23:38.026618 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.026583 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 10 02:23:38.026663 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.026639 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rzg95\"" Jul 10 02:23:38.026916 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.026903 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 10 02:23:38.083014 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.082987 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:38.083113 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.083021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:38.083113 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.083058 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009-signing-cabundle\") pod \"service-ca-79f68b9f4f-qcx9w\" (UID: \"cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qcx9w" Jul 10 02:23:38.083113 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.083105 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009-signing-key\") pod \"service-ca-79f68b9f4f-qcx9w\" (UID: \"cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qcx9w" Jul 10 02:23:38.083273 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:38.083117 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 02:23:38.083273 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:38.083127 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:23:38.083273 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.083143 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2579\" (UniqueName: \"kubernetes.io/projected/cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009-kube-api-access-s2579\") pod \"service-ca-79f68b9f4f-qcx9w\" (UID: \"cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qcx9w" Jul 10 02:23:38.083273 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:38.083181 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm podName:35865d4b-e324-4102-b2c6-0cfee7b6c10d nodeName:}" failed. No retries permitted until 2026-07-10 02:23:42.083161736 +0000 UTC m=+51.482614626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qzjnp" (UID: "35865d4b-e324-4102-b2c6-0cfee7b6c10d") : configmap "kube-rbac-proxy" not found Jul 10 02:23:38.083273 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:38.083220 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls podName:35865d4b-e324-4102-b2c6-0cfee7b6c10d nodeName:}" failed. No retries permitted until 2026-07-10 02:23:42.083195171 +0000 UTC m=+51.482648075 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qzjnp" (UID: "35865d4b-e324-4102-b2c6-0cfee7b6c10d") : secret "insights-runtime-extractor-tls" not found Jul 10 02:23:38.168529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.168510 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b79d4_ec1f23d1-33ad-495d-b737-ea19ddc98b69/serve-healthcheck-canary/0.log" Jul 10 02:23:38.184332 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.184309 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2579\" (UniqueName: \"kubernetes.io/projected/cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009-kube-api-access-s2579\") pod \"service-ca-79f68b9f4f-qcx9w\" (UID: \"cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qcx9w" Jul 10 02:23:38.184425 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.184404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009-signing-cabundle\") pod \"service-ca-79f68b9f4f-qcx9w\" (UID: \"cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qcx9w" Jul 10 02:23:38.184469 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.184456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009-signing-key\") pod \"service-ca-79f68b9f4f-qcx9w\" (UID: \"cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qcx9w" Jul 10 02:23:38.185123 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.185107 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009-signing-cabundle\") pod \"service-ca-79f68b9f4f-qcx9w\" (UID: \"cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qcx9w" Jul 10 02:23:38.186806 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.186788 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009-signing-key\") pod \"service-ca-79f68b9f4f-qcx9w\" (UID: \"cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qcx9w" Jul 10 02:23:38.192563 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.192545 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2579\" (UniqueName: \"kubernetes.io/projected/cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009-kube-api-access-s2579\") pod \"service-ca-79f68b9f4f-qcx9w\" (UID: \"cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qcx9w" Jul 10 02:23:38.328533 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.328469 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-qcx9w" Jul 10 02:23:38.483106 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:38.483074 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-qcx9w"] Jul 10 02:23:38.485804 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:23:38.485775 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc56c3d1_d8f5_46b2_aa4e_3e7f30b16009.slice/crio-a6694f8c3b6a0d4faadf1510f11ca74bde62054e2e14e5c38436a009849ae22c WatchSource:0}: Error finding container a6694f8c3b6a0d4faadf1510f11ca74bde62054e2e14e5c38436a009849ae22c: Status 404 returned error can't find the container with id a6694f8c3b6a0d4faadf1510f11ca74bde62054e2e14e5c38436a009849ae22c Jul 10 02:23:39.424445 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:39.424405 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-qcx9w" event={"ID":"cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009","Type":"ContainerStarted","Data":"a6694f8c3b6a0d4faadf1510f11ca74bde62054e2e14e5c38436a009849ae22c"} Jul 10 02:23:41.212261 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:41.212206 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:41.212618 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:41.212295 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:41.212618 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:41.212360 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:23:41.212618 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:41.212406 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 02:23:41.212618 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:41.212417 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6459bf6464-xcsdp: secret "image-registry-tls" not found Jul 10 02:23:41.212618 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:41.212429 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls podName:043938f8-88f9-438f-a386-a0bf37ef9b1f nodeName:}" failed. No retries permitted until 2026-07-10 02:23:57.212410755 +0000 UTC m=+66.611863669 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls") pod "dns-default-5drj5" (UID: "043938f8-88f9-438f-a386-a0bf37ef9b1f") : secret "dns-default-metrics-tls" not found Jul 10 02:23:41.212618 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:41.212449 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls podName:076c71ed-cbf4-4324-be05-ca3f80c1db93 nodeName:}" failed. No retries permitted until 2026-07-10 02:23:57.21243984 +0000 UTC m=+66.611892734 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls") pod "image-registry-6459bf6464-xcsdp" (UID: "076c71ed-cbf4-4324-be05-ca3f80c1db93") : secret "image-registry-tls" not found Jul 10 02:23:41.431420 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:41.431391 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-qcx9w" event={"ID":"cc56c3d1-d8f5-46b2-aa4e-3e7f30b16009","Type":"ContainerStarted","Data":"8f95843e2c56ac27d36ed485f1a4b7d37c0813cf2251eba0d594eaf6765c94e3"} Jul 10 02:23:41.450742 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:41.450698 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-qcx9w" podStartSLOduration=1.871881745 podStartE2EDuration="4.450684444s" totalImagesPullingTime="2.578802699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:41.449029865 +0000 UTC m=+50.848482776" watchObservedRunningTime="2026-07-10 02:23:41.450684444 +0000 UTC m=+50.850137357" Jul 10 02:23:42.119964 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:42.119931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:42.119964 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:42.119968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:42.120161 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:42.120043 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 02:23:42.120161 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:42.120070 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:23:42.120161 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:42.120090 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm podName:35865d4b-e324-4102-b2c6-0cfee7b6c10d nodeName:}" failed. No retries permitted until 2026-07-10 02:23:50.120076865 +0000 UTC m=+59.519529755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qzjnp" (UID: "35865d4b-e324-4102-b2c6-0cfee7b6c10d") : configmap "kube-rbac-proxy" not found Jul 10 02:23:42.120161 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:42.120125 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls podName:35865d4b-e324-4102-b2c6-0cfee7b6c10d nodeName:}" failed. No retries permitted until 2026-07-10 02:23:50.120111835 +0000 UTC m=+59.519564726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qzjnp" (UID: "35865d4b-e324-4102-b2c6-0cfee7b6c10d") : secret "insights-runtime-extractor-tls" not found Jul 10 02:23:44.137145 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:44.137109 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5b5xr\" (UID: \"7955372c-7076-4a1a-aff4-6cf9898d2576\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" Jul 10 02:23:44.139612 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:44.139579 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7955372c-7076-4a1a-aff4-6cf9898d2576-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5b5xr\" (UID: \"7955372c-7076-4a1a-aff4-6cf9898d2576\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" Jul 10 02:23:44.222641 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:44.222611 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" Jul 10 02:23:44.381880 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:44.381855 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr"] Jul 10 02:23:45.442745 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:45.442670 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" event={"ID":"7955372c-7076-4a1a-aff4-6cf9898d2576","Type":"ContainerStarted","Data":"2caf8b4e32a7d42d6f5d5d69a0c12c2ef775a092079fed38aef08eb38f88c4df"} Jul 10 02:23:46.447929 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:46.447893 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" event={"ID":"7955372c-7076-4a1a-aff4-6cf9898d2576","Type":"ContainerStarted","Data":"19519b123b531a117211d3d666bae1c065cba4b11d20444fdfa6efa6d6e44a22"} Jul 10 02:23:46.448296 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:46.447945 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" event={"ID":"7955372c-7076-4a1a-aff4-6cf9898d2576","Type":"ContainerStarted","Data":"33b96b95205b4c1aab6267f2433b38e0d4a2289d9969fbb4a39632d9e4663d6e"} Jul 10 02:23:46.468928 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:46.468887 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5b5xr" podStartSLOduration=16.651661761 podStartE2EDuration="18.468874129s" totalImagesPullingTime="1.817212368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:46.468695991 +0000 UTC m=+55.868148903" watchObservedRunningTime="2026-07-10 02:23:46.468874129 +0000 UTC m=+55.868327042" Jul 10 02:23:50.194117 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:50.194075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:50.194588 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:50.194114 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:50.194588 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:50.194219 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:23:50.194588 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:23:50.194301 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls podName:35865d4b-e324-4102-b2c6-0cfee7b6c10d nodeName:}" failed. No retries permitted until 2026-07-10 02:24:06.194285801 +0000 UTC m=+75.593738692 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qzjnp" (UID: "35865d4b-e324-4102-b2c6-0cfee7b6c10d") : secret "insights-runtime-extractor-tls" not found Jul 10 02:23:50.194694 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:50.194639 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35865d4b-e324-4102-b2c6-0cfee7b6c10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:23:52.382466 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:52.382433 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlm7j" Jul 10 02:23:55.837736 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:55.837694 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:55.841043 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:55.841021 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 02:23:55.850492 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:55.850467 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82138b66-c754-4c2d-a828-a104a92a9968-original-pull-secret\") pod \"global-pull-secret-syncer-lgw26\" (UID: \"82138b66-c754-4c2d-a828-a104a92a9968\") " pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:55.938698 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:55.938670 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs\") pod \"network-metrics-daemon-r4t9z\" (UID: \"90c11f6c-5762-4634-9dad-5dc5e5b751eb\") " pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:55.941607 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:55.941593 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 02:23:55.950919 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:55.950903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c11f6c-5762-4634-9dad-5dc5e5b751eb-metrics-certs\") pod \"network-metrics-daemon-r4t9z\" (UID: \"90c11f6c-5762-4634-9dad-5dc5e5b751eb\") " pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:56.039642 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.039619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnt5v\" (UniqueName: \"kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v\") pod \"network-check-target-j28gk\" (UID: \"e5e656b6-b264-455f-9da1-2de1fbac1a9f\") " pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:56.042678 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.042658 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 02:23:56.052720 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.052698 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 02:23:56.063283 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.063264 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnt5v\" (UniqueName: \"kubernetes.io/projected/e5e656b6-b264-455f-9da1-2de1fbac1a9f-kube-api-access-tnt5v\") pod \"network-check-target-j28gk\" (UID: \"e5e656b6-b264-455f-9da1-2de1fbac1a9f\") " pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:56.137347 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.137322 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gcczx\"" Jul 10 02:23:56.143339 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.143319 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgw26" Jul 10 02:23:56.145439 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.145424 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4t9z" Jul 10 02:23:56.238436 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.238415 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4r7lb\"" Jul 10 02:23:56.246526 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.246498 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:23:56.273496 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.273466 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r4t9z"] Jul 10 02:23:56.276726 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:23:56.276683 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c11f6c_5762_4634_9dad_5dc5e5b751eb.slice/crio-c3b8a4e6bf0f105d9b57921974c56f2e91e93734c0336b5edbb242f998bbccba WatchSource:0}: Error finding container c3b8a4e6bf0f105d9b57921974c56f2e91e93734c0336b5edbb242f998bbccba: Status 404 returned error can't find the container with id c3b8a4e6bf0f105d9b57921974c56f2e91e93734c0336b5edbb242f998bbccba Jul 10 02:23:56.291949 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.291924 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lgw26"] Jul 10 02:23:56.295140 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:23:56.295113 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82138b66_c754_4c2d_a828_a104a92a9968.slice/crio-cec49e426fdbf5430da15bcf93298a1bfb8b8ae7a26caf57a102784ef5da2ac3 WatchSource:0}: Error finding container cec49e426fdbf5430da15bcf93298a1bfb8b8ae7a26caf57a102784ef5da2ac3: Status 404 returned error can't find the container with id cec49e426fdbf5430da15bcf93298a1bfb8b8ae7a26caf57a102784ef5da2ac3 Jul 10 02:23:56.370782 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.370754 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j28gk"] Jul 10 02:23:56.373929 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:23:56.373905 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e656b6_b264_455f_9da1_2de1fbac1a9f.slice/crio-08b5ddb7dedf332139903798ea71f668007ac493ca3b2f9e015501a881331344 WatchSource:0}: Error finding container 08b5ddb7dedf332139903798ea71f668007ac493ca3b2f9e015501a881331344: Status 404 returned error can't find the container with id 08b5ddb7dedf332139903798ea71f668007ac493ca3b2f9e015501a881331344 Jul 10 02:23:56.478537 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.478462 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lgw26" event={"ID":"82138b66-c754-4c2d-a828-a104a92a9968","Type":"ContainerStarted","Data":"cec49e426fdbf5430da15bcf93298a1bfb8b8ae7a26caf57a102784ef5da2ac3"} Jul 10 02:23:56.479442 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.479417 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j28gk" event={"ID":"e5e656b6-b264-455f-9da1-2de1fbac1a9f","Type":"ContainerStarted","Data":"08b5ddb7dedf332139903798ea71f668007ac493ca3b2f9e015501a881331344"} Jul 10 02:23:56.480349 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:56.480332 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r4t9z" event={"ID":"90c11f6c-5762-4634-9dad-5dc5e5b751eb","Type":"ContainerStarted","Data":"c3b8a4e6bf0f105d9b57921974c56f2e91e93734c0336b5edbb242f998bbccba"} Jul 10 02:23:57.252988 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:57.252431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:57.252988 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:57.252508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:57.255844 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:57.255813 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/043938f8-88f9-438f-a386-a0bf37ef9b1f-metrics-tls\") pod \"dns-default-5drj5\" (UID: \"043938f8-88f9-438f-a386-a0bf37ef9b1f\") " pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:57.258647 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:57.258599 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls\") pod \"image-registry-6459bf6464-xcsdp\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:57.486115 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:57.486081 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r4t9z" event={"ID":"90c11f6c-5762-4634-9dad-5dc5e5b751eb","Type":"ContainerStarted","Data":"a8842d2740cd2a8fe8fe47a91b6af92dcf58f3e99cffeebfd959ac66b6fe59a9"} Jul 10 02:23:57.508547 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:57.508446 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zf7mn\"" Jul 10 02:23:57.516230 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:57.516206 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:57.526199 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:57.526171 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qdhtc\"" Jul 10 02:23:57.533610 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:57.533588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5drj5" Jul 10 02:23:57.690532 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:57.690225 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6459bf6464-xcsdp"] Jul 10 02:23:57.693201 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:23:57.693167 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076c71ed_cbf4_4324_be05_ca3f80c1db93.slice/crio-9ca955d4dcc8b71645fc573585b713114182d598b7c98b5a57ee587ef389d494 WatchSource:0}: Error finding container 9ca955d4dcc8b71645fc573585b713114182d598b7c98b5a57ee587ef389d494: Status 404 returned error can't find the container with id 9ca955d4dcc8b71645fc573585b713114182d598b7c98b5a57ee587ef389d494 Jul 10 02:23:57.701803 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:57.701783 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5drj5"] Jul 10 02:23:57.706622 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:23:57.706595 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043938f8_88f9_438f_a386_a0bf37ef9b1f.slice/crio-f48ca978a02c23ea1a59e8ac20f5d901b9ef45a6fa92b79bc5ce547a79a1f780 WatchSource:0}: Error finding container f48ca978a02c23ea1a59e8ac20f5d901b9ef45a6fa92b79bc5ce547a79a1f780: Status 404 returned error can't find the container with id f48ca978a02c23ea1a59e8ac20f5d901b9ef45a6fa92b79bc5ce547a79a1f780 Jul 10 02:23:58.493168 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:58.493115 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r4t9z" event={"ID":"90c11f6c-5762-4634-9dad-5dc5e5b751eb","Type":"ContainerStarted","Data":"99284b630d7e4f6b2f5a9cbe040ad1dee911900c22587e0234ac3c3ffcc4a25e"} Jul 10 02:23:58.495070 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:58.495038 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5drj5" event={"ID":"043938f8-88f9-438f-a386-a0bf37ef9b1f","Type":"ContainerStarted","Data":"f48ca978a02c23ea1a59e8ac20f5d901b9ef45a6fa92b79bc5ce547a79a1f780"} Jul 10 02:23:58.497449 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:58.497422 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" event={"ID":"076c71ed-cbf4-4324-be05-ca3f80c1db93","Type":"ContainerStarted","Data":"0a012128326a64bbed288f05c9724c5064e9c52e42dc8e69d32a61c1d5398603"} Jul 10 02:23:58.497577 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:58.497468 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" event={"ID":"076c71ed-cbf4-4324-be05-ca3f80c1db93","Type":"ContainerStarted","Data":"9ca955d4dcc8b71645fc573585b713114182d598b7c98b5a57ee587ef389d494"} Jul 10 02:23:58.497577 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:58.497516 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:23:58.512972 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:58.512918 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r4t9z" podStartSLOduration=66.407614719 podStartE2EDuration="1m7.51290005s" totalImagesPullingTime="1.105285331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:58.512742637 +0000 UTC m=+67.912195553" watchObservedRunningTime="2026-07-10 02:23:58.51290005 +0000 UTC m=+67.912353177" Jul 10 02:23:58.535105 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:23:58.535049 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" podStartSLOduration=69.53502768 podStartE2EDuration="1m9.53502768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:23:58.534518958 +0000 UTC m=+67.933971872" watchObservedRunningTime="2026-07-10 02:23:58.53502768 +0000 UTC m=+67.934480595" Jul 10 02:24:01.881954 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.881915 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc"] Jul 10 02:24:01.884176 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.884157 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:01.887184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.887167 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 02:24:01.888070 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.888055 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 02:24:01.888130 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.888092 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 02:24:01.888881 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.888860 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 02:24:01.888974 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.888860 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 02:24:01.889454 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.889438 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 02:24:01.889532 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.889490 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 02:24:01.903661 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.903636 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc"] Jul 10 02:24:01.990308 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.990274 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7a1108c-19ab-4a52-a435-aab6cab3a78c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:01.990469 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.990315 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7a1108c-19ab-4a52-a435-aab6cab3a78c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:01.990469 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.990355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfldg\" (UniqueName: \"kubernetes.io/projected/e7a1108c-19ab-4a52-a435-aab6cab3a78c-kube-api-access-cfldg\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:01.990469 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.990409 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a1108c-19ab-4a52-a435-aab6cab3a78c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:01.990469 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.990438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7a1108c-19ab-4a52-a435-aab6cab3a78c-hub\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:01.990597 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:01.990529 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7a1108c-19ab-4a52-a435-aab6cab3a78c-ca\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:02.091615 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.091576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7a1108c-19ab-4a52-a435-aab6cab3a78c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:02.091796 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.091624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfldg\" (UniqueName: \"kubernetes.io/projected/e7a1108c-19ab-4a52-a435-aab6cab3a78c-kube-api-access-cfldg\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:02.091796 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.091662 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a1108c-19ab-4a52-a435-aab6cab3a78c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:02.091796 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.091696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7a1108c-19ab-4a52-a435-aab6cab3a78c-hub\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:02.091965 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.091851 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7a1108c-19ab-4a52-a435-aab6cab3a78c-ca\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:02.091965 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.091916 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7a1108c-19ab-4a52-a435-aab6cab3a78c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:02.092387 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.092368 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a1108c-19ab-4a52-a435-aab6cab3a78c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:02.094177 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.094150 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7a1108c-19ab-4a52-a435-aab6cab3a78c-hub\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:02.094411 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.094395 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7a1108c-19ab-4a52-a435-aab6cab3a78c-ca\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:02.094464 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.094430 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7a1108c-19ab-4a52-a435-aab6cab3a78c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:02.094497 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.094460 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7a1108c-19ab-4a52-a435-aab6cab3a78c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:02.103023 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.103004 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfldg\" (UniqueName: \"kubernetes.io/projected/e7a1108c-19ab-4a52-a435-aab6cab3a78c-kube-api-access-cfldg\") pod \"cluster-proxy-proxy-agent-7c9567b78f-76dmc\" (UID: \"e7a1108c-19ab-4a52-a435-aab6cab3a78c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:02.201295 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.201171 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" Jul 10 02:24:02.329127 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.329049 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc"] Jul 10 02:24:02.331622 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:24:02.331589 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7a1108c_19ab_4a52_a435_aab6cab3a78c.slice/crio-55064e1df4c53d700aed6b15ef00b978e6b4753cb89800b8d5f645aa299b3c2c WatchSource:0}: Error finding container 55064e1df4c53d700aed6b15ef00b978e6b4753cb89800b8d5f645aa299b3c2c: Status 404 returned error can't find the container with id 55064e1df4c53d700aed6b15ef00b978e6b4753cb89800b8d5f645aa299b3c2c Jul 10 02:24:02.513065 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.512984 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lgw26" event={"ID":"82138b66-c754-4c2d-a828-a104a92a9968","Type":"ContainerStarted","Data":"c22de5a6d1535874f1e25dbd2309258089a0c99dcde54309454614d3b5266e2c"} Jul 10 02:24:02.514228 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.514205 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" event={"ID":"e7a1108c-19ab-4a52-a435-aab6cab3a78c","Type":"ContainerStarted","Data":"55064e1df4c53d700aed6b15ef00b978e6b4753cb89800b8d5f645aa299b3c2c"} Jul 10 02:24:02.515799 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.515769 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j28gk" event={"ID":"e5e656b6-b264-455f-9da1-2de1fbac1a9f","Type":"ContainerStarted","Data":"1bf54c71e0c4d69ad13f8c24cc0704c0a26b7c53d6bf6012733ab78fa175710b"} Jul 10 02:24:02.515947 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.515932 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:24:02.517545 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.517523 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5drj5" event={"ID":"043938f8-88f9-438f-a386-a0bf37ef9b1f","Type":"ContainerStarted","Data":"d5a9d644ac0c8b1b7ca9519329a3ab72be09b0114569fa50ffced61dfbb4f38f"} Jul 10 02:24:02.517637 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.517569 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5drj5" event={"ID":"043938f8-88f9-438f-a386-a0bf37ef9b1f","Type":"ContainerStarted","Data":"f7aa5e15aa57b21d5c63ca282723c5a1fe2d4cb4a12a66719c12a4fc44c397a7"} Jul 10 02:24:02.517637 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.517606 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5drj5" Jul 10 02:24:02.531671 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.531612 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lgw26" podStartSLOduration=66.322417738 podStartE2EDuration="1m11.53159949s" totalImagesPullingTime="5.209181752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:24:02.530262105 +0000 UTC m=+71.929715008" watchObservedRunningTime="2026-07-10 02:24:02.53159949 +0000 UTC m=+71.931052425" Jul 10 02:24:02.551173 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.551136 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5drj5" podStartSLOduration=33.761860372 podStartE2EDuration="37.551124561s" totalImagesPullingTime="3.789264189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:24:02.549991285 +0000 UTC m=+71.949444198" watchObservedRunningTime="2026-07-10 02:24:02.551124561 +0000 UTC m=+71.950577474" Jul 10 02:24:02.569025 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:02.568985 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j28gk" podStartSLOduration=66.449983092 podStartE2EDuration="1m11.56897226s" totalImagesPullingTime="5.118989168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:24:02.567978443 +0000 UTC m=+71.967431355" watchObservedRunningTime="2026-07-10 02:24:02.56897226 +0000 UTC m=+71.968425173" Jul 10 02:24:06.225456 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:06.225416 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:24:06.227688 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:06.227663 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35865d4b-e324-4102-b2c6-0cfee7b6c10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzjnp\" (UID: \"35865d4b-e324-4102-b2c6-0cfee7b6c10d\") " pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:24:06.234837 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:06.234811 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pfnlm\"" Jul 10 02:24:06.242279 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:06.242262 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qzjnp" Jul 10 02:24:06.529011 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:06.528969 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qzjnp"] Jul 10 02:24:06.532109 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:24:06.532083 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35865d4b_e324_4102_b2c6_0cfee7b6c10d.slice/crio-5a8397438d995ccdd6408a39edf38107230e4a17751fdea04acb00524e8d51ad WatchSource:0}: Error finding container 5a8397438d995ccdd6408a39edf38107230e4a17751fdea04acb00524e8d51ad: Status 404 returned error can't find the container with id 5a8397438d995ccdd6408a39edf38107230e4a17751fdea04acb00524e8d51ad Jul 10 02:24:06.535866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:06.535841 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" event={"ID":"e7a1108c-19ab-4a52-a435-aab6cab3a78c","Type":"ContainerStarted","Data":"fd1c262157950f9f90c9b2cc16d0cdf93d2543e7b937466235999d158cbe4262"} Jul 10 02:24:06.535949 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:06.535880 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" event={"ID":"e7a1108c-19ab-4a52-a435-aab6cab3a78c","Type":"ContainerStarted","Data":"38094084141fd2731804715635bd6273f3e78eaa7d00162bbb88de739d63fece"} Jul 10 02:24:07.542164 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:07.542118 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qzjnp" event={"ID":"35865d4b-e324-4102-b2c6-0cfee7b6c10d","Type":"ContainerStarted","Data":"5a8397438d995ccdd6408a39edf38107230e4a17751fdea04acb00524e8d51ad"} Jul 10 02:24:07.545227 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:07.545157 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" event={"ID":"e7a1108c-19ab-4a52-a435-aab6cab3a78c","Type":"ContainerStarted","Data":"c93593698744688b9515a92225cdcc566592e51531f8f0ef75212cec769afa24"} Jul 10 02:24:10.557589 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:10.557551 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qzjnp" event={"ID":"35865d4b-e324-4102-b2c6-0cfee7b6c10d","Type":"ContainerStarted","Data":"9471953c6cea3bc40cb1d89b11dc2327418431383a79b741e96c22e8be5190b3"} Jul 10 02:24:11.563623 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:11.563561 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qzjnp" event={"ID":"35865d4b-e324-4102-b2c6-0cfee7b6c10d","Type":"ContainerStarted","Data":"8e1ede248a48ec735c66ad8bf505e914957ee3beae095af9cbf6a994a8a780e0"} Jul 10 02:24:12.525435 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:12.525400 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5drj5" Jul 10 02:24:12.545604 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:12.545526 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c9567b78f-76dmc" podStartSLOduration=7.4795087 podStartE2EDuration="11.54550988s" totalImagesPullingTime="4.06600118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:24:07.571210444 +0000 UTC m=+76.970663361" watchObservedRunningTime="2026-07-10 02:24:12.54550988 +0000 UTC m=+81.944962793" Jul 10 02:24:13.572966 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:13.572937 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qzjnp" event={"ID":"35865d4b-e324-4102-b2c6-0cfee7b6c10d","Type":"ContainerStarted","Data":"44ed1e1f7b8780b30213a931bf5ef9ee2bae9696293af8cab3724191721cdc35"} Jul 10 02:24:13.595526 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:13.595483 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qzjnp" podStartSLOduration=32.750434943 podStartE2EDuration="39.595466918s" totalImagesPullingTime="6.845031975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:23:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:24:13.593577631 +0000 UTC m=+82.993030544" watchObservedRunningTime="2026-07-10 02:24:13.595466918 +0000 UTC m=+82.994919831" Jul 10 02:24:14.760823 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.760784 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lxnqq"] Jul 10 02:24:14.764471 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.764453 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.767204 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.767186 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-d22qb\"" Jul 10 02:24:14.767384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.767271 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 02:24:14.768370 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.768353 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 02:24:14.768528 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.768370 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 02:24:14.768715 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.768391 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 02:24:14.768834 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.768439 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 02:24:14.768936 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.768440 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 02:24:14.785225 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.785199 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e149b8-86a5-4801-a0df-84b5a98148c8-sys\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.785352 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.785237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6j42\" (UniqueName: \"kubernetes.io/projected/66e149b8-86a5-4801-a0df-84b5a98148c8-kube-api-access-n6j42\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.785352 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.785282 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-wtmp\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.785352 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.785311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-textfile\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.785508 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.785399 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-tls\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.785508 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.785463 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66e149b8-86a5-4801-a0df-84b5a98148c8-root\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.785508 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.785500 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.785658 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.785525 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.785658 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.785561 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66e149b8-86a5-4801-a0df-84b5a98148c8-metrics-client-ca\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.886541 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.886509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e149b8-86a5-4801-a0df-84b5a98148c8-sys\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.886703 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.886554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6j42\" (UniqueName: \"kubernetes.io/projected/66e149b8-86a5-4801-a0df-84b5a98148c8-kube-api-access-n6j42\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.886703 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.886589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-wtmp\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.886703 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.886592 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e149b8-86a5-4801-a0df-84b5a98148c8-sys\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.886703 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.886621 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-textfile\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.886703 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.886647 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-tls\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.886703 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.886689 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66e149b8-86a5-4801-a0df-84b5a98148c8-root\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.887018 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.886722 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.887018 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.886741 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-wtmp\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.887018 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.886746 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.887018 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.886780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66e149b8-86a5-4801-a0df-84b5a98148c8-metrics-client-ca\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.887018 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:24:14.886847 2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 02:24:14.887018 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:24:14.886904 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-tls podName:66e149b8-86a5-4801-a0df-84b5a98148c8 nodeName:}" failed. No retries permitted until 2026-07-10 02:24:15.386881123 +0000 UTC m=+84.786334019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-tls") pod "node-exporter-lxnqq" (UID: "66e149b8-86a5-4801-a0df-84b5a98148c8") : secret "node-exporter-tls" not found Jul 10 02:24:14.887356 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.887223 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-textfile\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.887356 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.887299 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66e149b8-86a5-4801-a0df-84b5a98148c8-root\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.887473 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.887452 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66e149b8-86a5-4801-a0df-84b5a98148c8-metrics-client-ca\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.887748 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.887722 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.890092 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.890069 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:14.895817 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:14.895796 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6j42\" (UniqueName: \"kubernetes.io/projected/66e149b8-86a5-4801-a0df-84b5a98148c8-kube-api-access-n6j42\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:15.389975 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:15.389938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-tls\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:15.392231 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:15.392198 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66e149b8-86a5-4801-a0df-84b5a98148c8-node-exporter-tls\") pod \"node-exporter-lxnqq\" (UID: \"66e149b8-86a5-4801-a0df-84b5a98148c8\") " pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:15.674257 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:15.674172 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lxnqq" Jul 10 02:24:15.685293 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:24:15.685260 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e149b8_86a5_4801_a0df_84b5a98148c8.slice/crio-860892ac9202871dddc424cd2aa6adbe36dcfd4ef5eff692e6277fee282a1a24 WatchSource:0}: Error finding container 860892ac9202871dddc424cd2aa6adbe36dcfd4ef5eff692e6277fee282a1a24: Status 404 returned error can't find the container with id 860892ac9202871dddc424cd2aa6adbe36dcfd4ef5eff692e6277fee282a1a24 Jul 10 02:24:16.584850 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:16.584816 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lxnqq" event={"ID":"66e149b8-86a5-4801-a0df-84b5a98148c8","Type":"ContainerStarted","Data":"860892ac9202871dddc424cd2aa6adbe36dcfd4ef5eff692e6277fee282a1a24"} Jul 10 02:24:17.520854 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:17.520819 2575 patch_prober.go:28] interesting pod/image-registry-6459bf6464-xcsdp container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 02:24:17.521003 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:17.520872 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" podUID="076c71ed-cbf4-4324-be05-ca3f80c1db93" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 02:24:17.591853 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:17.591823 2575 generic.go:356] "Generic (PLEG): container finished" podID="66e149b8-86a5-4801-a0df-84b5a98148c8" containerID="92fc430786c537d6d8356e1c320037fa3be64f9f61c0a7e0781a87ac28e8a861" exitCode=0 Jul 10 02:24:17.592264 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:17.591899 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lxnqq" event={"ID":"66e149b8-86a5-4801-a0df-84b5a98148c8","Type":"ContainerDied","Data":"92fc430786c537d6d8356e1c320037fa3be64f9f61c0a7e0781a87ac28e8a861"} Jul 10 02:24:18.597164 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:18.597127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lxnqq" event={"ID":"66e149b8-86a5-4801-a0df-84b5a98148c8","Type":"ContainerStarted","Data":"0449f961898dcd2b08d459ebe1d429b793bc3aea883cc37283b3069676348ca7"} Jul 10 02:24:18.597576 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:18.597181 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lxnqq" event={"ID":"66e149b8-86a5-4801-a0df-84b5a98148c8","Type":"ContainerStarted","Data":"05f78b0e0016160551b915b5a5ed3df3a44a7e768416d3b14ff3bcd586cf3428"} Jul 10 02:24:18.623016 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:18.622972 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lxnqq" podStartSLOduration=3.446670316 podStartE2EDuration="4.622957269s" totalImagesPullingTime="1.112916132s" totalInitContainerRuntime="63.370821ms" isStatefulPod=true podCreationTimestamp="2026-07-10 02:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:24:18.621130215 +0000 UTC m=+88.020583129" watchObservedRunningTime="2026-07-10 02:24:18.622957269 +0000 UTC m=+88.022410181" Jul 10 02:24:19.185878 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.185844 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc"] Jul 10 02:24:19.187964 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.187949 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.191927 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.191903 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 02:24:19.192062 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.191987 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wr8gh\"" Jul 10 02:24:19.192062 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.191992 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 02:24:19.192174 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.192080 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 02:24:19.192232 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.192212 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 02:24:19.192307 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.192255 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7ut16nk9ppinq\"" Jul 10 02:24:19.200908 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.200890 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc"] Jul 10 02:24:19.218446 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.218411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c9ef7c30-6c22-4d92-b998-1d94ce678b25-client-ca-bundle\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.218585 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.218502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c9ef7c30-6c22-4d92-b998-1d94ce678b25-metrics-server-audit-profiles\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.218585 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.218537 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c9ef7c30-6c22-4d92-b998-1d94ce678b25-secret-metrics-server-tls\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.218706 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.218626 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9ef7c30-6c22-4d92-b998-1d94ce678b25-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.218706 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.218692 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c9ef7c30-6c22-4d92-b998-1d94ce678b25-secret-metrics-server-client-certs\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.218798 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.218715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c9ef7c30-6c22-4d92-b998-1d94ce678b25-audit-log\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.218798 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.218737 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mcd9\" (UniqueName: \"kubernetes.io/projected/c9ef7c30-6c22-4d92-b998-1d94ce678b25-kube-api-access-9mcd9\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.319870 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.319841 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9ef7c30-6c22-4d92-b998-1d94ce678b25-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.320021 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.319881 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c9ef7c30-6c22-4d92-b998-1d94ce678b25-secret-metrics-server-client-certs\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.320021 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.319901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c9ef7c30-6c22-4d92-b998-1d94ce678b25-audit-log\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.320021 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.320005 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mcd9\" (UniqueName: \"kubernetes.io/projected/c9ef7c30-6c22-4d92-b998-1d94ce678b25-kube-api-access-9mcd9\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.320172 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.320064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c9ef7c30-6c22-4d92-b998-1d94ce678b25-client-ca-bundle\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.320172 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.320117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c9ef7c30-6c22-4d92-b998-1d94ce678b25-metrics-server-audit-profiles\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.320172 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.320150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c9ef7c30-6c22-4d92-b998-1d94ce678b25-secret-metrics-server-tls\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.320460 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.320441 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c9ef7c30-6c22-4d92-b998-1d94ce678b25-audit-log\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.320637 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.320618 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9ef7c30-6c22-4d92-b998-1d94ce678b25-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.321103 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.321078 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c9ef7c30-6c22-4d92-b998-1d94ce678b25-metrics-server-audit-profiles\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.322528 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.322507 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c9ef7c30-6c22-4d92-b998-1d94ce678b25-client-ca-bundle\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.322634 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.322558 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c9ef7c30-6c22-4d92-b998-1d94ce678b25-secret-metrics-server-client-certs\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.322683 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.322631 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c9ef7c30-6c22-4d92-b998-1d94ce678b25-secret-metrics-server-tls\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.328952 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.328927 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mcd9\" (UniqueName: \"kubernetes.io/projected/c9ef7c30-6c22-4d92-b998-1d94ce678b25-kube-api-access-9mcd9\") pod \"metrics-server-6bcbfc5cb6-flcgc\" (UID: \"c9ef7c30-6c22-4d92-b998-1d94ce678b25\") " pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.497024 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.496947 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:19.506028 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.506003 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:24:19.649061 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:19.649036 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc"] Jul 10 02:24:19.651482 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:24:19.651458 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ef7c30_6c22_4d92_b998_1d94ce678b25.slice/crio-175bae66f4e32ba2d01c91149982ba166e6a6d6cba5441698774864d0a4e0ce0 WatchSource:0}: Error finding container 175bae66f4e32ba2d01c91149982ba166e6a6d6cba5441698774864d0a4e0ce0: Status 404 returned error can't find the container with id 175bae66f4e32ba2d01c91149982ba166e6a6d6cba5441698774864d0a4e0ce0 Jul 10 02:24:20.605454 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:20.605420 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" event={"ID":"c9ef7c30-6c22-4d92-b998-1d94ce678b25","Type":"ContainerStarted","Data":"175bae66f4e32ba2d01c91149982ba166e6a6d6cba5441698774864d0a4e0ce0"} Jul 10 02:24:21.612965 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:21.612931 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" event={"ID":"c9ef7c30-6c22-4d92-b998-1d94ce678b25","Type":"ContainerStarted","Data":"376b0d490a62a3ea3302b0521f196f864e5db085cc9cc99e2405b8545fc92a04"} Jul 10 02:24:21.645773 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:21.645717 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" podStartSLOduration=1.17098445 podStartE2EDuration="2.645699619s" totalImagesPullingTime="1.474715169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:24:21.643324325 +0000 UTC m=+91.042777237" watchObservedRunningTime="2026-07-10 02:24:21.645699619 +0000 UTC m=+91.045152538" Jul 10 02:24:25.604124 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:25.604092 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6459bf6464-xcsdp"] Jul 10 02:24:33.525335 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:33.525303 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j28gk" Jul 10 02:24:39.497331 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:39.497296 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:39.497776 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:39.497357 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:44.288980 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.288945 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-788b56947c-xvvlz"] Jul 10 02:24:44.417610 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.417573 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-788b56947c-xvvlz"] Jul 10 02:24:44.417763 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.417666 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.420688 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.420663 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 02:24:44.420795 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.420663 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 02:24:44.421814 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.421798 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 02:24:44.421936 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.421875 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 02:24:44.421936 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.421901 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 02:24:44.422119 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.422106 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 02:24:44.422492 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.422472 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 02:24:44.422576 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.422552 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5l4mr\"" Jul 10 02:24:44.427496 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.427475 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 02:24:44.519314 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.519277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-service-ca\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.519462 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.519331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d71cd387-c324-4490-bb21-0e155ffd847c-console-serving-cert\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.519462 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.519386 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-oauth-serving-cert\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.519462 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.519421 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-console-config\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.519462 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.519437 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbx8n\" (UniqueName: \"kubernetes.io/projected/d71cd387-c324-4490-bb21-0e155ffd847c-kube-api-access-cbx8n\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.519601 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.519468 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d71cd387-c324-4490-bb21-0e155ffd847c-console-oauth-config\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.519601 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.519493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-trusted-ca-bundle\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.620445 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.620404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d71cd387-c324-4490-bb21-0e155ffd847c-console-serving-cert\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.620630 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.620459 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-oauth-serving-cert\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.620630 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.620492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-console-config\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.620630 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.620515 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbx8n\" (UniqueName: \"kubernetes.io/projected/d71cd387-c324-4490-bb21-0e155ffd847c-kube-api-access-cbx8n\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.620630 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.620562 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d71cd387-c324-4490-bb21-0e155ffd847c-console-oauth-config\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.620630 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.620586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-trusted-ca-bundle\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.620859 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.620632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-service-ca\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.621428 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.621400 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-oauth-serving-cert\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.621556 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.621429 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-console-config\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.621624 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.621553 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-service-ca\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.621624 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.621615 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-trusted-ca-bundle\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.622991 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.622965 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d71cd387-c324-4490-bb21-0e155ffd847c-console-serving-cert\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.623101 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.623081 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d71cd387-c324-4490-bb21-0e155ffd847c-console-oauth-config\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.644797 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.644765 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbx8n\" (UniqueName: \"kubernetes.io/projected/d71cd387-c324-4490-bb21-0e155ffd847c-kube-api-access-cbx8n\") pod \"console-788b56947c-xvvlz\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.727106 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.727071 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:44.876174 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:44.876103 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-788b56947c-xvvlz"] Jul 10 02:24:44.879006 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:24:44.878982 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71cd387_c324_4490_bb21_0e155ffd847c.slice/crio-e51b7b3a5c13e20a991a88880075769d15ac2326a2d109ea79fa94e8a26c0bd7 WatchSource:0}: Error finding container e51b7b3a5c13e20a991a88880075769d15ac2326a2d109ea79fa94e8a26c0bd7: Status 404 returned error can't find the container with id e51b7b3a5c13e20a991a88880075769d15ac2326a2d109ea79fa94e8a26c0bd7 Jul 10 02:24:45.694416 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:45.694367 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788b56947c-xvvlz" event={"ID":"d71cd387-c324-4490-bb21-0e155ffd847c","Type":"ContainerStarted","Data":"e51b7b3a5c13e20a991a88880075769d15ac2326a2d109ea79fa94e8a26c0bd7"} Jul 10 02:24:47.703295 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:47.703238 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788b56947c-xvvlz" event={"ID":"d71cd387-c324-4490-bb21-0e155ffd847c","Type":"ContainerStarted","Data":"bc8e9972488ca7dda0799aeda2b48e98ddf859fcf757a3e94fea6f75c329863c"} Jul 10 02:24:47.733156 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:47.733106 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-788b56947c-xvvlz" podStartSLOduration=1.110272004 podStartE2EDuration="3.733090735s" totalImagesPullingTime="2.622818731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:24:47.732207253 +0000 UTC m=+117.131660167" watchObservedRunningTime="2026-07-10 02:24:47.733090735 +0000 UTC m=+117.132543648" Jul 10 02:24:50.629166 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.629099 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" podUID="076c71ed-cbf4-4324-be05-ca3f80c1db93" containerName="registry" containerID="cri-o://0a012128326a64bbed288f05c9724c5064e9c52e42dc8e69d32a61c1d5398603" gracePeriod=30 Jul 10 02:24:50.866413 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.866387 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:24:50.972196 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.972126 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-certificates\") pod \"076c71ed-cbf4-4324-be05-ca3f80c1db93\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " Jul 10 02:24:50.972196 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.972158 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/076c71ed-cbf4-4324-be05-ca3f80c1db93-trusted-ca\") pod \"076c71ed-cbf4-4324-be05-ca3f80c1db93\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " Jul 10 02:24:50.972196 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.972198 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29wsz\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-kube-api-access-29wsz\") pod \"076c71ed-cbf4-4324-be05-ca3f80c1db93\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " Jul 10 02:24:50.972471 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.972228 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/076c71ed-cbf4-4324-be05-ca3f80c1db93-image-registry-private-configuration\") pod \"076c71ed-cbf4-4324-be05-ca3f80c1db93\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " Jul 10 02:24:50.972471 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.972277 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-bound-sa-token\") pod \"076c71ed-cbf4-4324-be05-ca3f80c1db93\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " Jul 10 02:24:50.972471 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.972308 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/076c71ed-cbf4-4324-be05-ca3f80c1db93-installation-pull-secrets\") pod \"076c71ed-cbf4-4324-be05-ca3f80c1db93\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " Jul 10 02:24:50.972471 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.972432 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/076c71ed-cbf4-4324-be05-ca3f80c1db93-ca-trust-extracted\") pod \"076c71ed-cbf4-4324-be05-ca3f80c1db93\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " Jul 10 02:24:50.972678 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.972485 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls\") pod \"076c71ed-cbf4-4324-be05-ca3f80c1db93\" (UID: \"076c71ed-cbf4-4324-be05-ca3f80c1db93\") " Jul 10 02:24:50.972678 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.972659 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "076c71ed-cbf4-4324-be05-ca3f80c1db93" (UID: "076c71ed-cbf4-4324-be05-ca3f80c1db93"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:24:50.972770 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.972736 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076c71ed-cbf4-4324-be05-ca3f80c1db93-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "076c71ed-cbf4-4324-be05-ca3f80c1db93" (UID: "076c71ed-cbf4-4324-be05-ca3f80c1db93"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:24:50.972826 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.972813 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-certificates\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:24:50.972878 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.972834 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/076c71ed-cbf4-4324-be05-ca3f80c1db93-trusted-ca\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:24:50.974769 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.974720 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/076c71ed-cbf4-4324-be05-ca3f80c1db93-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "076c71ed-cbf4-4324-be05-ca3f80c1db93" (UID: "076c71ed-cbf4-4324-be05-ca3f80c1db93"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:24:50.974899 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.974821 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "076c71ed-cbf4-4324-be05-ca3f80c1db93" (UID: "076c71ed-cbf4-4324-be05-ca3f80c1db93"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:24:50.974899 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.974864 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/076c71ed-cbf4-4324-be05-ca3f80c1db93-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "076c71ed-cbf4-4324-be05-ca3f80c1db93" (UID: "076c71ed-cbf4-4324-be05-ca3f80c1db93"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:24:50.975013 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.974910 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-kube-api-access-29wsz" (OuterVolumeSpecName: "kube-api-access-29wsz") pod "076c71ed-cbf4-4324-be05-ca3f80c1db93" (UID: "076c71ed-cbf4-4324-be05-ca3f80c1db93"). InnerVolumeSpecName "kube-api-access-29wsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:24:50.975013 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.974996 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "076c71ed-cbf4-4324-be05-ca3f80c1db93" (UID: "076c71ed-cbf4-4324-be05-ca3f80c1db93"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:24:50.981401 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:50.981376 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076c71ed-cbf4-4324-be05-ca3f80c1db93-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "076c71ed-cbf4-4324-be05-ca3f80c1db93" (UID: "076c71ed-cbf4-4324-be05-ca3f80c1db93"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:24:51.073537 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.073504 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29wsz\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-kube-api-access-29wsz\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:24:51.073537 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.073528 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/076c71ed-cbf4-4324-be05-ca3f80c1db93-image-registry-private-configuration\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:24:51.073537 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.073538 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-bound-sa-token\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:24:51.073817 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.073547 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/076c71ed-cbf4-4324-be05-ca3f80c1db93-installation-pull-secrets\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:24:51.073817 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.073555 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/076c71ed-cbf4-4324-be05-ca3f80c1db93-ca-trust-extracted\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:24:51.073817 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.073564 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/076c71ed-cbf4-4324-be05-ca3f80c1db93-registry-tls\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:24:51.721228 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.721196 2575 generic.go:356] "Generic (PLEG): container finished" podID="076c71ed-cbf4-4324-be05-ca3f80c1db93" containerID="0a012128326a64bbed288f05c9724c5064e9c52e42dc8e69d32a61c1d5398603" exitCode=0 Jul 10 02:24:51.721669 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.721267 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" Jul 10 02:24:51.721669 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.721277 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" event={"ID":"076c71ed-cbf4-4324-be05-ca3f80c1db93","Type":"ContainerDied","Data":"0a012128326a64bbed288f05c9724c5064e9c52e42dc8e69d32a61c1d5398603"} Jul 10 02:24:51.721669 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.721352 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6459bf6464-xcsdp" event={"ID":"076c71ed-cbf4-4324-be05-ca3f80c1db93","Type":"ContainerDied","Data":"9ca955d4dcc8b71645fc573585b713114182d598b7c98b5a57ee587ef389d494"} Jul 10 02:24:51.721669 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.721380 2575 scope.go:117] "RemoveContainer" containerID="0a012128326a64bbed288f05c9724c5064e9c52e42dc8e69d32a61c1d5398603" Jul 10 02:24:51.729396 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.729380 2575 scope.go:117] "RemoveContainer" containerID="0a012128326a64bbed288f05c9724c5064e9c52e42dc8e69d32a61c1d5398603" Jul 10 02:24:51.729645 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:24:51.729625 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a012128326a64bbed288f05c9724c5064e9c52e42dc8e69d32a61c1d5398603\": container with ID starting with 0a012128326a64bbed288f05c9724c5064e9c52e42dc8e69d32a61c1d5398603 not found: ID does not exist" containerID="0a012128326a64bbed288f05c9724c5064e9c52e42dc8e69d32a61c1d5398603" Jul 10 02:24:51.729695 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.729656 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a012128326a64bbed288f05c9724c5064e9c52e42dc8e69d32a61c1d5398603"} err="failed to get container status \"0a012128326a64bbed288f05c9724c5064e9c52e42dc8e69d32a61c1d5398603\": rpc error: code = NotFound desc = could not find container \"0a012128326a64bbed288f05c9724c5064e9c52e42dc8e69d32a61c1d5398603\": container with ID starting with 0a012128326a64bbed288f05c9724c5064e9c52e42dc8e69d32a61c1d5398603 not found: ID does not exist" Jul 10 02:24:51.740685 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.740667 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6459bf6464-xcsdp"] Jul 10 02:24:51.745763 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:51.745743 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6459bf6464-xcsdp"] Jul 10 02:24:53.128260 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:53.128203 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="076c71ed-cbf4-4324-be05-ca3f80c1db93" path="/var/lib/kubelet/pods/076c71ed-cbf4-4324-be05-ca3f80c1db93/volumes" Jul 10 02:24:54.727356 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:54.727321 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:54.727743 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:54.727599 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:54.732387 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:54.732365 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:55.737969 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:55.737944 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:24:59.502841 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:59.502808 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:59.506444 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:59.506423 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6bcbfc5cb6-flcgc" Jul 10 02:24:59.748379 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:59.748343 2575 generic.go:356] "Generic (PLEG): container finished" podID="0ce04276-ac1b-463f-b812-5dd0b2c82145" containerID="5c0e9d75bbe91bd78f7b360fa6193eb8430ecabbe27278e861b5f393a397e5bd" exitCode=0 Jul 10 02:24:59.748526 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:59.748420 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" event={"ID":"0ce04276-ac1b-463f-b812-5dd0b2c82145","Type":"ContainerDied","Data":"5c0e9d75bbe91bd78f7b360fa6193eb8430ecabbe27278e861b5f393a397e5bd"} Jul 10 02:24:59.748888 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:24:59.748870 2575 scope.go:117] "RemoveContainer" containerID="5c0e9d75bbe91bd78f7b360fa6193eb8430ecabbe27278e861b5f393a397e5bd" Jul 10 02:25:00.753231 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:25:00.753193 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-fkdsw" event={"ID":"0ce04276-ac1b-463f-b812-5dd0b2c82145","Type":"ContainerStarted","Data":"aa0ad7cc85557854ddee3d0dc543d4829738362f248697ada135990a0cc5c602"} Jul 10 02:25:43.047724 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:25:43.047689 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-788b56947c-xvvlz"] Jul 10 02:26:08.067056 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.067013 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-788b56947c-xvvlz" podUID="d71cd387-c324-4490-bb21-0e155ffd847c" containerName="console" containerID="cri-o://bc8e9972488ca7dda0799aeda2b48e98ddf859fcf757a3e94fea6f75c329863c" gracePeriod=15 Jul 10 02:26:08.309322 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.309298 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:26:08.434890 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.434861 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-oauth-serving-cert\") pod \"d71cd387-c324-4490-bb21-0e155ffd847c\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " Jul 10 02:26:08.435072 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.434914 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-console-config\") pod \"d71cd387-c324-4490-bb21-0e155ffd847c\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " Jul 10 02:26:08.435072 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.434937 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-trusted-ca-bundle\") pod \"d71cd387-c324-4490-bb21-0e155ffd847c\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " Jul 10 02:26:08.435171 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.435067 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbx8n\" (UniqueName: \"kubernetes.io/projected/d71cd387-c324-4490-bb21-0e155ffd847c-kube-api-access-cbx8n\") pod \"d71cd387-c324-4490-bb21-0e155ffd847c\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " Jul 10 02:26:08.435171 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.435141 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d71cd387-c324-4490-bb21-0e155ffd847c-console-oauth-config\") pod \"d71cd387-c324-4490-bb21-0e155ffd847c\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " Jul 10 02:26:08.435241 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.435188 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-service-ca\") pod \"d71cd387-c324-4490-bb21-0e155ffd847c\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " Jul 10 02:26:08.435241 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.435235 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d71cd387-c324-4490-bb21-0e155ffd847c-console-serving-cert\") pod \"d71cd387-c324-4490-bb21-0e155ffd847c\" (UID: \"d71cd387-c324-4490-bb21-0e155ffd847c\") " Jul 10 02:26:08.435374 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.435347 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d71cd387-c324-4490-bb21-0e155ffd847c" (UID: "d71cd387-c324-4490-bb21-0e155ffd847c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:26:08.435435 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.435367 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-console-config" (OuterVolumeSpecName: "console-config") pod "d71cd387-c324-4490-bb21-0e155ffd847c" (UID: "d71cd387-c324-4490-bb21-0e155ffd847c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:26:08.435552 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.435529 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d71cd387-c324-4490-bb21-0e155ffd847c" (UID: "d71cd387-c324-4490-bb21-0e155ffd847c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:26:08.435636 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.435614 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-oauth-serving-cert\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:26:08.435636 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.435621 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-service-ca" (OuterVolumeSpecName: "service-ca") pod "d71cd387-c324-4490-bb21-0e155ffd847c" (UID: "d71cd387-c324-4490-bb21-0e155ffd847c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:26:08.435818 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.435637 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-console-config\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:26:08.435818 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.435686 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-trusted-ca-bundle\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:26:08.437369 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.437347 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d71cd387-c324-4490-bb21-0e155ffd847c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d71cd387-c324-4490-bb21-0e155ffd847c" (UID: "d71cd387-c324-4490-bb21-0e155ffd847c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:26:08.437451 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.437372 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d71cd387-c324-4490-bb21-0e155ffd847c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d71cd387-c324-4490-bb21-0e155ffd847c" (UID: "d71cd387-c324-4490-bb21-0e155ffd847c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:26:08.437451 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.437372 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71cd387-c324-4490-bb21-0e155ffd847c-kube-api-access-cbx8n" (OuterVolumeSpecName: "kube-api-access-cbx8n") pod "d71cd387-c324-4490-bb21-0e155ffd847c" (UID: "d71cd387-c324-4490-bb21-0e155ffd847c"). InnerVolumeSpecName "kube-api-access-cbx8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:26:08.536854 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.536814 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d71cd387-c324-4490-bb21-0e155ffd847c-console-oauth-config\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:26:08.536854 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.536849 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d71cd387-c324-4490-bb21-0e155ffd847c-service-ca\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:26:08.536854 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.536858 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d71cd387-c324-4490-bb21-0e155ffd847c-console-serving-cert\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:26:08.536854 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.536867 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbx8n\" (UniqueName: \"kubernetes.io/projected/d71cd387-c324-4490-bb21-0e155ffd847c-kube-api-access-cbx8n\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:26:08.984495 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.984457 2575 generic.go:356] "Generic (PLEG): container finished" podID="d71cd387-c324-4490-bb21-0e155ffd847c" containerID="bc8e9972488ca7dda0799aeda2b48e98ddf859fcf757a3e94fea6f75c329863c" exitCode=0 Jul 10 02:26:08.984660 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.984504 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788b56947c-xvvlz" Jul 10 02:26:08.984660 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.984515 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788b56947c-xvvlz" event={"ID":"d71cd387-c324-4490-bb21-0e155ffd847c","Type":"ContainerDied","Data":"bc8e9972488ca7dda0799aeda2b48e98ddf859fcf757a3e94fea6f75c329863c"} Jul 10 02:26:08.984660 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.984566 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788b56947c-xvvlz" event={"ID":"d71cd387-c324-4490-bb21-0e155ffd847c","Type":"ContainerDied","Data":"e51b7b3a5c13e20a991a88880075769d15ac2326a2d109ea79fa94e8a26c0bd7"} Jul 10 02:26:08.984660 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.984591 2575 scope.go:117] "RemoveContainer" containerID="bc8e9972488ca7dda0799aeda2b48e98ddf859fcf757a3e94fea6f75c329863c" Jul 10 02:26:08.992585 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.992567 2575 scope.go:117] "RemoveContainer" containerID="bc8e9972488ca7dda0799aeda2b48e98ddf859fcf757a3e94fea6f75c329863c" Jul 10 02:26:08.992843 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:26:08.992821 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc8e9972488ca7dda0799aeda2b48e98ddf859fcf757a3e94fea6f75c329863c\": container with ID starting with bc8e9972488ca7dda0799aeda2b48e98ddf859fcf757a3e94fea6f75c329863c not found: ID does not exist" containerID="bc8e9972488ca7dda0799aeda2b48e98ddf859fcf757a3e94fea6f75c329863c" Jul 10 02:26:08.992886 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:08.992853 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc8e9972488ca7dda0799aeda2b48e98ddf859fcf757a3e94fea6f75c329863c"} err="failed to get container status \"bc8e9972488ca7dda0799aeda2b48e98ddf859fcf757a3e94fea6f75c329863c\": rpc error: code = NotFound desc = could not find container \"bc8e9972488ca7dda0799aeda2b48e98ddf859fcf757a3e94fea6f75c329863c\": container with ID starting with bc8e9972488ca7dda0799aeda2b48e98ddf859fcf757a3e94fea6f75c329863c not found: ID does not exist" Jul 10 02:26:09.005763 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:09.005738 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-788b56947c-xvvlz"] Jul 10 02:26:09.009698 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:09.009680 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-788b56947c-xvvlz"] Jul 10 02:26:09.127957 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:09.127927 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d71cd387-c324-4490-bb21-0e155ffd847c" path="/var/lib/kubelet/pods/d71cd387-c324-4490-bb21-0e155ffd847c/volumes" Jul 10 02:26:11.854651 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.854611 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2"] Jul 10 02:26:11.855072 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.854854 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d71cd387-c324-4490-bb21-0e155ffd847c" containerName="console" Jul 10 02:26:11.855072 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.854865 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d71cd387-c324-4490-bb21-0e155ffd847c" containerName="console" Jul 10 02:26:11.855072 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.854880 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="076c71ed-cbf4-4324-be05-ca3f80c1db93" containerName="registry" Jul 10 02:26:11.855072 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.854885 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="076c71ed-cbf4-4324-be05-ca3f80c1db93" containerName="registry" Jul 10 02:26:11.855072 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.854927 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d71cd387-c324-4490-bb21-0e155ffd847c" containerName="console" Jul 10 02:26:11.855072 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.854934 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="076c71ed-cbf4-4324-be05-ca3f80c1db93" containerName="registry" Jul 10 02:26:11.858975 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.858958 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" Jul 10 02:26:11.861781 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.861759 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 02:26:11.862824 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.862803 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 02:26:11.862927 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.862803 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-phmc2\"" Jul 10 02:26:11.873075 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.873053 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2"] Jul 10 02:26:11.963069 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.963035 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bffbf5a-dd49-496a-acc6-9e7e755fe033-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2\" (UID: \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" Jul 10 02:26:11.963069 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.963081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bffbf5a-dd49-496a-acc6-9e7e755fe033-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2\" (UID: \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" Jul 10 02:26:11.963331 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:11.963127 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h62rm\" (UniqueName: \"kubernetes.io/projected/3bffbf5a-dd49-496a-acc6-9e7e755fe033-kube-api-access-h62rm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2\" (UID: \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" Jul 10 02:26:12.063814 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:12.063781 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bffbf5a-dd49-496a-acc6-9e7e755fe033-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2\" (UID: \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" Jul 10 02:26:12.063966 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:12.063824 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bffbf5a-dd49-496a-acc6-9e7e755fe033-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2\" (UID: \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" Jul 10 02:26:12.063966 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:12.063869 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h62rm\" (UniqueName: \"kubernetes.io/projected/3bffbf5a-dd49-496a-acc6-9e7e755fe033-kube-api-access-h62rm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2\" (UID: \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" Jul 10 02:26:12.064179 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:12.064157 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bffbf5a-dd49-496a-acc6-9e7e755fe033-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2\" (UID: \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" Jul 10 02:26:12.064281 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:12.064219 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bffbf5a-dd49-496a-acc6-9e7e755fe033-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2\" (UID: \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" Jul 10 02:26:12.073727 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:12.073702 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h62rm\" (UniqueName: \"kubernetes.io/projected/3bffbf5a-dd49-496a-acc6-9e7e755fe033-kube-api-access-h62rm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2\" (UID: \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" Jul 10 02:26:12.167061 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:12.166972 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" Jul 10 02:26:12.292423 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:12.292394 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2"] Jul 10 02:26:12.294741 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:26:12.294711 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bffbf5a_dd49_496a_acc6_9e7e755fe033.slice/crio-c34eb594262ea44fa64aba4fc35f6e2609db56d2b5d7451421116ea5753ae693 WatchSource:0}: Error finding container c34eb594262ea44fa64aba4fc35f6e2609db56d2b5d7451421116ea5753ae693: Status 404 returned error can't find the container with id c34eb594262ea44fa64aba4fc35f6e2609db56d2b5d7451421116ea5753ae693 Jul 10 02:26:13.005425 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:13.005389 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" event={"ID":"3bffbf5a-dd49-496a-acc6-9e7e755fe033","Type":"ContainerStarted","Data":"c34eb594262ea44fa64aba4fc35f6e2609db56d2b5d7451421116ea5753ae693"} Jul 10 02:26:20.032059 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:20.032025 2575 generic.go:356] "Generic (PLEG): container finished" podID="3bffbf5a-dd49-496a-acc6-9e7e755fe033" containerID="a07a1a0a591e9bbf95a424f7f794c193699991bc00e32fd24d4c1a70ef83c1fc" exitCode=0 Jul 10 02:26:20.032460 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:20.032074 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" event={"ID":"3bffbf5a-dd49-496a-acc6-9e7e755fe033","Type":"ContainerDied","Data":"a07a1a0a591e9bbf95a424f7f794c193699991bc00e32fd24d4c1a70ef83c1fc"} Jul 10 02:26:23.044710 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:23.044671 2575 generic.go:356] "Generic (PLEG): container finished" podID="3bffbf5a-dd49-496a-acc6-9e7e755fe033" containerID="d0ae68bca7c5a4f34d32c6c77b9aeb34b4ff8595ac3295b8f64ac581eb740e39" exitCode=0 Jul 10 02:26:23.045083 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:23.044762 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" event={"ID":"3bffbf5a-dd49-496a-acc6-9e7e755fe033","Type":"ContainerDied","Data":"d0ae68bca7c5a4f34d32c6c77b9aeb34b4ff8595ac3295b8f64ac581eb740e39"} Jul 10 02:26:31.080758 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:31.080727 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" event={"ID":"3bffbf5a-dd49-496a-acc6-9e7e755fe033","Type":"ContainerStarted","Data":"1d9e5a858f61517b206337b79085e380cad7dfe0513c062ef5b4201ac78d4754"} Jul 10 02:26:31.107781 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:31.107723 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" podStartSLOduration=3.320062543 podStartE2EDuration="20.107702163s" totalImagesPullingTime="16.78763962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:26:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:26:31.107048659 +0000 UTC m=+220.506501589" watchObservedRunningTime="2026-07-10 02:26:31.107702163 +0000 UTC m=+220.507155079" Jul 10 02:26:32.086730 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:32.086699 2575 generic.go:356] "Generic (PLEG): container finished" podID="3bffbf5a-dd49-496a-acc6-9e7e755fe033" containerID="1d9e5a858f61517b206337b79085e380cad7dfe0513c062ef5b4201ac78d4754" exitCode=0 Jul 10 02:26:32.087114 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:32.086793 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" event={"ID":"3bffbf5a-dd49-496a-acc6-9e7e755fe033","Type":"ContainerDied","Data":"1d9e5a858f61517b206337b79085e380cad7dfe0513c062ef5b4201ac78d4754"} Jul 10 02:26:33.234593 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:33.234569 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" Jul 10 02:26:33.334427 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:33.334390 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bffbf5a-dd49-496a-acc6-9e7e755fe033-bundle\") pod \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\" (UID: \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\") " Jul 10 02:26:33.334607 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:33.334458 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h62rm\" (UniqueName: \"kubernetes.io/projected/3bffbf5a-dd49-496a-acc6-9e7e755fe033-kube-api-access-h62rm\") pod \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\" (UID: \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\") " Jul 10 02:26:33.334607 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:33.334487 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bffbf5a-dd49-496a-acc6-9e7e755fe033-util\") pod \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\" (UID: \"3bffbf5a-dd49-496a-acc6-9e7e755fe033\") " Jul 10 02:26:33.335092 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:33.335062 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bffbf5a-dd49-496a-acc6-9e7e755fe033-bundle" (OuterVolumeSpecName: "bundle") pod "3bffbf5a-dd49-496a-acc6-9e7e755fe033" (UID: "3bffbf5a-dd49-496a-acc6-9e7e755fe033"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:26:33.336633 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:33.336601 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bffbf5a-dd49-496a-acc6-9e7e755fe033-kube-api-access-h62rm" (OuterVolumeSpecName: "kube-api-access-h62rm") pod "3bffbf5a-dd49-496a-acc6-9e7e755fe033" (UID: "3bffbf5a-dd49-496a-acc6-9e7e755fe033"). InnerVolumeSpecName "kube-api-access-h62rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:26:33.339160 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:33.339136 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bffbf5a-dd49-496a-acc6-9e7e755fe033-util" (OuterVolumeSpecName: "util") pod "3bffbf5a-dd49-496a-acc6-9e7e755fe033" (UID: "3bffbf5a-dd49-496a-acc6-9e7e755fe033"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:26:33.435846 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:33.435815 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h62rm\" (UniqueName: \"kubernetes.io/projected/3bffbf5a-dd49-496a-acc6-9e7e755fe033-kube-api-access-h62rm\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:26:33.435846 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:33.435842 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bffbf5a-dd49-496a-acc6-9e7e755fe033-util\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:26:33.435846 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:33.435852 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bffbf5a-dd49-496a-acc6-9e7e755fe033-bundle\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:26:34.096671 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:34.096642 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" Jul 10 02:26:34.096671 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:34.096645 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xdjc2" event={"ID":"3bffbf5a-dd49-496a-acc6-9e7e755fe033","Type":"ContainerDied","Data":"c34eb594262ea44fa64aba4fc35f6e2609db56d2b5d7451421116ea5753ae693"} Jul 10 02:26:34.096866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:34.096697 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c34eb594262ea44fa64aba4fc35f6e2609db56d2b5d7451421116ea5753ae693" Jul 10 02:26:40.129577 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.129545 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hgpnm"] Jul 10 02:26:40.130040 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.129783 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bffbf5a-dd49-496a-acc6-9e7e755fe033" containerName="util" Jul 10 02:26:40.130040 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.129794 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bffbf5a-dd49-496a-acc6-9e7e755fe033" containerName="util" Jul 10 02:26:40.130040 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.129814 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bffbf5a-dd49-496a-acc6-9e7e755fe033" containerName="pull" Jul 10 02:26:40.130040 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.129819 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bffbf5a-dd49-496a-acc6-9e7e755fe033" containerName="pull" Jul 10 02:26:40.130040 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.129825 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bffbf5a-dd49-496a-acc6-9e7e755fe033" containerName="extract" Jul 10 02:26:40.130040 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.129831 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bffbf5a-dd49-496a-acc6-9e7e755fe033" containerName="extract" Jul 10 02:26:40.130040 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.129865 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bffbf5a-dd49-496a-acc6-9e7e755fe033" containerName="extract" Jul 10 02:26:40.132918 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.132903 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hgpnm" Jul 10 02:26:40.136762 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.136740 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 10 02:26:40.136878 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.136829 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vbrxl\"" Jul 10 02:26:40.136970 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.136954 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:26:40.149995 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.149974 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hgpnm"] Jul 10 02:26:40.183345 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.183317 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvvxd\" (UniqueName: \"kubernetes.io/projected/a65e1d00-23e3-473d-b8f2-935c7db1a349-kube-api-access-qvvxd\") pod \"cert-manager-operator-controller-manager-55c444bf86-hgpnm\" (UID: \"a65e1d00-23e3-473d-b8f2-935c7db1a349\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hgpnm" Jul 10 02:26:40.183445 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.183362 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a65e1d00-23e3-473d-b8f2-935c7db1a349-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hgpnm\" (UID: \"a65e1d00-23e3-473d-b8f2-935c7db1a349\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hgpnm" Jul 10 02:26:40.284240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.284215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a65e1d00-23e3-473d-b8f2-935c7db1a349-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hgpnm\" (UID: \"a65e1d00-23e3-473d-b8f2-935c7db1a349\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hgpnm" Jul 10 02:26:40.284352 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.284294 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvxd\" (UniqueName: \"kubernetes.io/projected/a65e1d00-23e3-473d-b8f2-935c7db1a349-kube-api-access-qvvxd\") pod \"cert-manager-operator-controller-manager-55c444bf86-hgpnm\" (UID: \"a65e1d00-23e3-473d-b8f2-935c7db1a349\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hgpnm" Jul 10 02:26:40.284584 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.284568 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a65e1d00-23e3-473d-b8f2-935c7db1a349-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hgpnm\" (UID: \"a65e1d00-23e3-473d-b8f2-935c7db1a349\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hgpnm" Jul 10 02:26:40.297740 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.297713 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvvxd\" (UniqueName: \"kubernetes.io/projected/a65e1d00-23e3-473d-b8f2-935c7db1a349-kube-api-access-qvvxd\") pod \"cert-manager-operator-controller-manager-55c444bf86-hgpnm\" (UID: \"a65e1d00-23e3-473d-b8f2-935c7db1a349\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hgpnm" Jul 10 02:26:40.441588 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.441509 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hgpnm" Jul 10 02:26:40.576410 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:40.576388 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hgpnm"] Jul 10 02:26:40.578930 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:26:40.578899 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65e1d00_23e3_473d_b8f2_935c7db1a349.slice/crio-f83269973d46315209bac8d7f0f262f9231d5446caf269eeba55571b1c380a4a WatchSource:0}: Error finding container f83269973d46315209bac8d7f0f262f9231d5446caf269eeba55571b1c380a4a: Status 404 returned error can't find the container with id f83269973d46315209bac8d7f0f262f9231d5446caf269eeba55571b1c380a4a Jul 10 02:26:41.122193 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:41.122159 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hgpnm" event={"ID":"a65e1d00-23e3-473d-b8f2-935c7db1a349","Type":"ContainerStarted","Data":"f83269973d46315209bac8d7f0f262f9231d5446caf269eeba55571b1c380a4a"} Jul 10 02:26:43.132609 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:43.132575 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hgpnm" event={"ID":"a65e1d00-23e3-473d-b8f2-935c7db1a349","Type":"ContainerStarted","Data":"0997cae81d9a8109322d9d164788b1e89671dbfcb47946dc37f07ceb3e04f2a6"} Jul 10 02:26:43.161671 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:43.161619 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hgpnm" podStartSLOduration=1.5323973309999999 podStartE2EDuration="3.161593295s" totalImagesPullingTime="1.629195964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:26:43.159640169 +0000 UTC m=+232.559093082" watchObservedRunningTime="2026-07-10 02:26:43.161593295 +0000 UTC m=+232.561046208" Jul 10 02:26:43.839473 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:43.839440 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4"] Jul 10 02:26:43.842404 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:43.842388 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" Jul 10 02:26:43.847067 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:43.847044 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 02:26:43.847172 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:43.847138 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-phmc2\"" Jul 10 02:26:43.848349 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:43.848329 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 02:26:43.878874 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:43.878849 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4"] Jul 10 02:26:43.916028 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:43.915998 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16ff34fc-bb93-4611-866e-6f52d964b75d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4\" (UID: \"16ff34fc-bb93-4611-866e-6f52d964b75d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" Jul 10 02:26:43.916185 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:43.916036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2krfn\" (UniqueName: \"kubernetes.io/projected/16ff34fc-bb93-4611-866e-6f52d964b75d-kube-api-access-2krfn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4\" (UID: \"16ff34fc-bb93-4611-866e-6f52d964b75d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" Jul 10 02:26:43.916185 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:43.916068 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16ff34fc-bb93-4611-866e-6f52d964b75d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4\" (UID: \"16ff34fc-bb93-4611-866e-6f52d964b75d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" Jul 10 02:26:44.016434 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:44.016396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16ff34fc-bb93-4611-866e-6f52d964b75d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4\" (UID: \"16ff34fc-bb93-4611-866e-6f52d964b75d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" Jul 10 02:26:44.016625 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:44.016439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2krfn\" (UniqueName: \"kubernetes.io/projected/16ff34fc-bb93-4611-866e-6f52d964b75d-kube-api-access-2krfn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4\" (UID: \"16ff34fc-bb93-4611-866e-6f52d964b75d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" Jul 10 02:26:44.016625 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:44.016469 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16ff34fc-bb93-4611-866e-6f52d964b75d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4\" (UID: \"16ff34fc-bb93-4611-866e-6f52d964b75d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" Jul 10 02:26:44.016855 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:44.016834 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16ff34fc-bb93-4611-866e-6f52d964b75d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4\" (UID: \"16ff34fc-bb93-4611-866e-6f52d964b75d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" Jul 10 02:26:44.016911 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:44.016887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16ff34fc-bb93-4611-866e-6f52d964b75d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4\" (UID: \"16ff34fc-bb93-4611-866e-6f52d964b75d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" Jul 10 02:26:44.040806 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:44.040775 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2krfn\" (UniqueName: \"kubernetes.io/projected/16ff34fc-bb93-4611-866e-6f52d964b75d-kube-api-access-2krfn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4\" (UID: \"16ff34fc-bb93-4611-866e-6f52d964b75d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" Jul 10 02:26:44.151739 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:44.151710 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" Jul 10 02:26:44.297538 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:44.297510 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4"] Jul 10 02:26:44.299967 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:26:44.299934 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ff34fc_bb93_4611_866e_6f52d964b75d.slice/crio-4c0d783e90a482fb9098bf699a41147145f9f9825225c96eacf79dde963fb8da WatchSource:0}: Error finding container 4c0d783e90a482fb9098bf699a41147145f9f9825225c96eacf79dde963fb8da: Status 404 returned error can't find the container with id 4c0d783e90a482fb9098bf699a41147145f9f9825225c96eacf79dde963fb8da Jul 10 02:26:45.147659 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:45.147615 2575 generic.go:356] "Generic (PLEG): container finished" podID="16ff34fc-bb93-4611-866e-6f52d964b75d" containerID="28a4e23d63cae4462b2e277eaf43b1f422bb0d86556c251ba4549a821bbfe087" exitCode=0 Jul 10 02:26:45.147816 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:45.147674 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" event={"ID":"16ff34fc-bb93-4611-866e-6f52d964b75d","Type":"ContainerDied","Data":"28a4e23d63cae4462b2e277eaf43b1f422bb0d86556c251ba4549a821bbfe087"} Jul 10 02:26:45.147816 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:45.147734 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" event={"ID":"16ff34fc-bb93-4611-866e-6f52d964b75d","Type":"ContainerStarted","Data":"4c0d783e90a482fb9098bf699a41147145f9f9825225c96eacf79dde963fb8da"} Jul 10 02:26:46.861690 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:46.861651 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-s67dg"] Jul 10 02:26:46.869856 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:46.869837 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-s67dg" Jul 10 02:26:46.872658 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:46.872637 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 02:26:46.872771 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:46.872639 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vtk26\"" Jul 10 02:26:46.873876 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:46.873847 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 02:26:46.876634 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:46.876611 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-s67dg"] Jul 10 02:26:46.942184 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:46.942154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkznr\" (UniqueName: \"kubernetes.io/projected/3713a93c-c260-4caf-b3a4-0beed317befb-kube-api-access-fkznr\") pod \"cert-manager-cainjector-559cdd8c87-s67dg\" (UID: \"3713a93c-c260-4caf-b3a4-0beed317befb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-s67dg" Jul 10 02:26:46.942334 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:46.942191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3713a93c-c260-4caf-b3a4-0beed317befb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-s67dg\" (UID: \"3713a93c-c260-4caf-b3a4-0beed317befb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-s67dg" Jul 10 02:26:47.042970 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:47.042936 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkznr\" (UniqueName: \"kubernetes.io/projected/3713a93c-c260-4caf-b3a4-0beed317befb-kube-api-access-fkznr\") pod \"cert-manager-cainjector-559cdd8c87-s67dg\" (UID: \"3713a93c-c260-4caf-b3a4-0beed317befb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-s67dg" Jul 10 02:26:47.042970 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:47.042973 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3713a93c-c260-4caf-b3a4-0beed317befb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-s67dg\" (UID: \"3713a93c-c260-4caf-b3a4-0beed317befb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-s67dg" Jul 10 02:26:47.062432 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:47.062404 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkznr\" (UniqueName: \"kubernetes.io/projected/3713a93c-c260-4caf-b3a4-0beed317befb-kube-api-access-fkznr\") pod \"cert-manager-cainjector-559cdd8c87-s67dg\" (UID: \"3713a93c-c260-4caf-b3a4-0beed317befb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-s67dg" Jul 10 02:26:47.063207 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:47.063190 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3713a93c-c260-4caf-b3a4-0beed317befb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-s67dg\" (UID: \"3713a93c-c260-4caf-b3a4-0beed317befb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-s67dg" Jul 10 02:26:47.158141 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:47.158060 2575 generic.go:356] "Generic (PLEG): container finished" podID="16ff34fc-bb93-4611-866e-6f52d964b75d" containerID="a7dd3126152bfdd0054109b07fb74f83aa5f0586b26858d0e2932f6b9004ed68" exitCode=0 Jul 10 02:26:47.158141 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:47.158132 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" event={"ID":"16ff34fc-bb93-4611-866e-6f52d964b75d","Type":"ContainerDied","Data":"a7dd3126152bfdd0054109b07fb74f83aa5f0586b26858d0e2932f6b9004ed68"} Jul 10 02:26:47.179669 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:47.179647 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-s67dg" Jul 10 02:26:47.313957 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:47.313929 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-s67dg"] Jul 10 02:26:47.317012 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:26:47.316985 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3713a93c_c260_4caf_b3a4_0beed317befb.slice/crio-c37abc6f7af696f9ef846a93fff83a1341316a316f84313412389cc650e61844 WatchSource:0}: Error finding container c37abc6f7af696f9ef846a93fff83a1341316a316f84313412389cc650e61844: Status 404 returned error can't find the container with id c37abc6f7af696f9ef846a93fff83a1341316a316f84313412389cc650e61844 Jul 10 02:26:48.164083 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:48.164039 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-s67dg" event={"ID":"3713a93c-c260-4caf-b3a4-0beed317befb","Type":"ContainerStarted","Data":"c37abc6f7af696f9ef846a93fff83a1341316a316f84313412389cc650e61844"} Jul 10 02:26:48.166544 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:48.166497 2575 generic.go:356] "Generic (PLEG): container finished" podID="16ff34fc-bb93-4611-866e-6f52d964b75d" containerID="8ab8b5422127a9ccb1c7e481b1a80917e9dfdea122449df0f203b09babfa75f8" exitCode=0 Jul 10 02:26:48.166684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:48.166547 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" event={"ID":"16ff34fc-bb93-4611-866e-6f52d964b75d","Type":"ContainerDied","Data":"8ab8b5422127a9ccb1c7e481b1a80917e9dfdea122449df0f203b09babfa75f8"} Jul 10 02:26:49.969306 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:49.969281 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" Jul 10 02:26:50.069433 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:50.069382 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2krfn\" (UniqueName: \"kubernetes.io/projected/16ff34fc-bb93-4611-866e-6f52d964b75d-kube-api-access-2krfn\") pod \"16ff34fc-bb93-4611-866e-6f52d964b75d\" (UID: \"16ff34fc-bb93-4611-866e-6f52d964b75d\") " Jul 10 02:26:50.069571 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:50.069467 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16ff34fc-bb93-4611-866e-6f52d964b75d-bundle\") pod \"16ff34fc-bb93-4611-866e-6f52d964b75d\" (UID: \"16ff34fc-bb93-4611-866e-6f52d964b75d\") " Jul 10 02:26:50.069571 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:50.069527 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16ff34fc-bb93-4611-866e-6f52d964b75d-util\") pod \"16ff34fc-bb93-4611-866e-6f52d964b75d\" (UID: \"16ff34fc-bb93-4611-866e-6f52d964b75d\") " Jul 10 02:26:50.070219 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:50.070190 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ff34fc-bb93-4611-866e-6f52d964b75d-bundle" (OuterVolumeSpecName: "bundle") pod "16ff34fc-bb93-4611-866e-6f52d964b75d" (UID: "16ff34fc-bb93-4611-866e-6f52d964b75d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:26:50.071510 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:50.071485 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ff34fc-bb93-4611-866e-6f52d964b75d-kube-api-access-2krfn" (OuterVolumeSpecName: "kube-api-access-2krfn") pod "16ff34fc-bb93-4611-866e-6f52d964b75d" (UID: "16ff34fc-bb93-4611-866e-6f52d964b75d"). InnerVolumeSpecName "kube-api-access-2krfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:26:50.074781 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:50.074757 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ff34fc-bb93-4611-866e-6f52d964b75d-util" (OuterVolumeSpecName: "util") pod "16ff34fc-bb93-4611-866e-6f52d964b75d" (UID: "16ff34fc-bb93-4611-866e-6f52d964b75d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:26:50.170850 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:50.170816 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16ff34fc-bb93-4611-866e-6f52d964b75d-bundle\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:26:50.170850 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:50.170841 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16ff34fc-bb93-4611-866e-6f52d964b75d-util\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:26:50.170850 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:50.170853 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2krfn\" (UniqueName: \"kubernetes.io/projected/16ff34fc-bb93-4611-866e-6f52d964b75d-kube-api-access-2krfn\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:26:50.179762 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:50.179721 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-s67dg" event={"ID":"3713a93c-c260-4caf-b3a4-0beed317befb","Type":"ContainerStarted","Data":"c5d80b056ed75f4208a9ac2a12e8427423eb7a20f40a344f82eea9bd1d198a39"} Jul 10 02:26:50.181563 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:50.181539 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" event={"ID":"16ff34fc-bb93-4611-866e-6f52d964b75d","Type":"ContainerDied","Data":"4c0d783e90a482fb9098bf699a41147145f9f9825225c96eacf79dde963fb8da"} Jul 10 02:26:50.181718 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:50.181584 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c0d783e90a482fb9098bf699a41147145f9f9825225c96eacf79dde963fb8da" Jul 10 02:26:50.181718 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:50.181592 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6jr4" Jul 10 02:26:51.031589 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:51.031470 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-s67dg" podStartSLOduration=2.355886624 podStartE2EDuration="5.031450472s" totalImagesPullingTime="2.675563848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:26:50.210848955 +0000 UTC m=+239.610301867" watchObservedRunningTime="2026-07-10 02:26:51.031450472 +0000 UTC m=+240.430903385" Jul 10 02:26:57.152885 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.152850 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24sgd"] Jul 10 02:26:57.153292 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.153101 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16ff34fc-bb93-4611-866e-6f52d964b75d" containerName="util" Jul 10 02:26:57.153292 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.153113 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="16ff34fc-bb93-4611-866e-6f52d964b75d" containerName="util" Jul 10 02:26:57.153292 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.153121 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16ff34fc-bb93-4611-866e-6f52d964b75d" containerName="pull" Jul 10 02:26:57.153292 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.153127 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="16ff34fc-bb93-4611-866e-6f52d964b75d" containerName="pull" Jul 10 02:26:57.153292 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.153133 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16ff34fc-bb93-4611-866e-6f52d964b75d" containerName="extract" Jul 10 02:26:57.153292 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.153139 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="16ff34fc-bb93-4611-866e-6f52d964b75d" containerName="extract" Jul 10 02:26:57.153292 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.153181 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="16ff34fc-bb93-4611-866e-6f52d964b75d" containerName="extract" Jul 10 02:26:57.162617 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.162596 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24sgd" Jul 10 02:26:57.167859 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.167837 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 02:26:57.169031 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.169010 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:26:57.169031 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.169019 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-5f9hg\"" Jul 10 02:26:57.172209 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.172187 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24sgd"] Jul 10 02:26:57.237770 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.237738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55fzc\" (UniqueName: \"kubernetes.io/projected/965ee09b-bb61-4294-865c-32f3429f8c72-kube-api-access-55fzc\") pod \"openshift-lws-operator-fd8ccff4c-24sgd\" (UID: \"965ee09b-bb61-4294-865c-32f3429f8c72\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24sgd" Jul 10 02:26:57.238103 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.238089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/965ee09b-bb61-4294-865c-32f3429f8c72-tmp\") pod \"openshift-lws-operator-fd8ccff4c-24sgd\" (UID: \"965ee09b-bb61-4294-865c-32f3429f8c72\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24sgd" Jul 10 02:26:57.339063 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.339030 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/965ee09b-bb61-4294-865c-32f3429f8c72-tmp\") pod \"openshift-lws-operator-fd8ccff4c-24sgd\" (UID: \"965ee09b-bb61-4294-865c-32f3429f8c72\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24sgd" Jul 10 02:26:57.339205 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.339097 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55fzc\" (UniqueName: \"kubernetes.io/projected/965ee09b-bb61-4294-865c-32f3429f8c72-kube-api-access-55fzc\") pod \"openshift-lws-operator-fd8ccff4c-24sgd\" (UID: \"965ee09b-bb61-4294-865c-32f3429f8c72\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24sgd" Jul 10 02:26:57.339445 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.339426 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/965ee09b-bb61-4294-865c-32f3429f8c72-tmp\") pod \"openshift-lws-operator-fd8ccff4c-24sgd\" (UID: \"965ee09b-bb61-4294-865c-32f3429f8c72\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24sgd" Jul 10 02:26:57.347958 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.347925 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55fzc\" (UniqueName: \"kubernetes.io/projected/965ee09b-bb61-4294-865c-32f3429f8c72-kube-api-access-55fzc\") pod \"openshift-lws-operator-fd8ccff4c-24sgd\" (UID: \"965ee09b-bb61-4294-865c-32f3429f8c72\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24sgd" Jul 10 02:26:57.471925 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.471849 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24sgd" Jul 10 02:26:57.626427 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:57.626402 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24sgd"] Jul 10 02:26:57.628340 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:26:57.628311 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod965ee09b_bb61_4294_865c_32f3429f8c72.slice/crio-0477dee1333ff5a536cc7dba65b14568099e2e211c6fb2f1505c506dbca2913d WatchSource:0}: Error finding container 0477dee1333ff5a536cc7dba65b14568099e2e211c6fb2f1505c506dbca2913d: Status 404 returned error can't find the container with id 0477dee1333ff5a536cc7dba65b14568099e2e211c6fb2f1505c506dbca2913d Jul 10 02:26:58.211860 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:26:58.211826 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24sgd" event={"ID":"965ee09b-bb61-4294-865c-32f3429f8c72","Type":"ContainerStarted","Data":"0477dee1333ff5a536cc7dba65b14568099e2e211c6fb2f1505c506dbca2913d"} Jul 10 02:27:01.225593 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:01.225557 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24sgd" event={"ID":"965ee09b-bb61-4294-865c-32f3429f8c72","Type":"ContainerStarted","Data":"7c9e19d07685f01cac0ae29de10010bc35e530be4135b732bb964d71278ac432"} Jul 10 02:27:01.248057 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:01.248008 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24sgd" podStartSLOduration=1.296698502 podStartE2EDuration="4.247988021s" totalImagesPullingTime="2.951289519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:27:01.245883453 +0000 UTC m=+250.645336365" watchObservedRunningTime="2026-07-10 02:27:01.247988021 +0000 UTC m=+250.647440933" Jul 10 02:27:03.708904 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:03.708870 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw"] Jul 10 02:27:03.711420 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:03.711403 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" Jul 10 02:27:03.714199 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:03.714180 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 02:27:03.714344 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:03.714222 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-phmc2\"" Jul 10 02:27:03.715376 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:03.715359 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 02:27:03.722083 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:03.722059 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw"] Jul 10 02:27:03.896405 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:03.896373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw\" (UID: \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" Jul 10 02:27:03.896586 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:03.896408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4j2g\" (UniqueName: \"kubernetes.io/projected/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-kube-api-access-m4j2g\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw\" (UID: \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" Jul 10 02:27:03.896586 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:03.896440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw\" (UID: \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" Jul 10 02:27:03.997646 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:03.997538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw\" (UID: \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" Jul 10 02:27:03.997646 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:03.997598 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j2g\" (UniqueName: \"kubernetes.io/projected/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-kube-api-access-m4j2g\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw\" (UID: \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" Jul 10 02:27:03.997646 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:03.997649 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw\" (UID: \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" Jul 10 02:27:03.998011 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:03.997996 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw\" (UID: \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" Jul 10 02:27:03.998048 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:03.998033 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw\" (UID: \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" Jul 10 02:27:04.006936 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:04.006911 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4j2g\" (UniqueName: \"kubernetes.io/projected/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-kube-api-access-m4j2g\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw\" (UID: \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" Jul 10 02:27:04.019884 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:04.019860 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" Jul 10 02:27:04.144786 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:04.144763 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw"] Jul 10 02:27:04.147357 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:27:04.147331 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefe18e7_ab0f_4eca_9ced_6714a53c3c94.slice/crio-5eb3df377da0dd1d080c16801728b5541c7d806bb7d3bb0b73809b1589e31bbb WatchSource:0}: Error finding container 5eb3df377da0dd1d080c16801728b5541c7d806bb7d3bb0b73809b1589e31bbb: Status 404 returned error can't find the container with id 5eb3df377da0dd1d080c16801728b5541c7d806bb7d3bb0b73809b1589e31bbb Jul 10 02:27:04.238600 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:04.238552 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" event={"ID":"aefe18e7-ab0f-4eca-9ced-6714a53c3c94","Type":"ContainerStarted","Data":"9db03b13d12429fd2b8b4311ca15c18153b06180d1c1071f93e0d7661704d41a"} Jul 10 02:27:04.238765 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:04.238615 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" event={"ID":"aefe18e7-ab0f-4eca-9ced-6714a53c3c94","Type":"ContainerStarted","Data":"5eb3df377da0dd1d080c16801728b5541c7d806bb7d3bb0b73809b1589e31bbb"} Jul 10 02:27:05.244234 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:05.244200 2575 generic.go:356] "Generic (PLEG): container finished" podID="aefe18e7-ab0f-4eca-9ced-6714a53c3c94" containerID="9db03b13d12429fd2b8b4311ca15c18153b06180d1c1071f93e0d7661704d41a" exitCode=0 Jul 10 02:27:05.244616 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:05.244284 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" event={"ID":"aefe18e7-ab0f-4eca-9ced-6714a53c3c94","Type":"ContainerDied","Data":"9db03b13d12429fd2b8b4311ca15c18153b06180d1c1071f93e0d7661704d41a"} Jul 10 02:27:06.250337 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:06.250302 2575 generic.go:356] "Generic (PLEG): container finished" podID="aefe18e7-ab0f-4eca-9ced-6714a53c3c94" containerID="ccbf24f0a93cff066f5e22f8e237193e75270a521a8c9940e83bc94a4c8a5287" exitCode=0 Jul 10 02:27:06.250706 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:06.250383 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" event={"ID":"aefe18e7-ab0f-4eca-9ced-6714a53c3c94","Type":"ContainerDied","Data":"ccbf24f0a93cff066f5e22f8e237193e75270a521a8c9940e83bc94a4c8a5287"} Jul 10 02:27:07.256005 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:07.255971 2575 generic.go:356] "Generic (PLEG): container finished" podID="aefe18e7-ab0f-4eca-9ced-6714a53c3c94" containerID="8e3da170965ee49d5b76b44d14e985f91027026913d78fce3910f554d2d8a795" exitCode=0 Jul 10 02:27:07.256395 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:07.256036 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" event={"ID":"aefe18e7-ab0f-4eca-9ced-6714a53c3c94","Type":"ContainerDied","Data":"8e3da170965ee49d5b76b44d14e985f91027026913d78fce3910f554d2d8a795"} Jul 10 02:27:08.377893 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:08.377864 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" Jul 10 02:27:08.532751 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:08.532672 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-util\") pod \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\" (UID: \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\") " Jul 10 02:27:08.532751 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:08.532719 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4j2g\" (UniqueName: \"kubernetes.io/projected/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-kube-api-access-m4j2g\") pod \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\" (UID: \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\") " Jul 10 02:27:08.532928 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:08.532840 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-bundle\") pod \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\" (UID: \"aefe18e7-ab0f-4eca-9ced-6714a53c3c94\") " Jul 10 02:27:08.533658 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:08.533632 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-bundle" (OuterVolumeSpecName: "bundle") pod "aefe18e7-ab0f-4eca-9ced-6714a53c3c94" (UID: "aefe18e7-ab0f-4eca-9ced-6714a53c3c94"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:27:08.534773 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:08.534750 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-kube-api-access-m4j2g" (OuterVolumeSpecName: "kube-api-access-m4j2g") pod "aefe18e7-ab0f-4eca-9ced-6714a53c3c94" (UID: "aefe18e7-ab0f-4eca-9ced-6714a53c3c94"). InnerVolumeSpecName "kube-api-access-m4j2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:27:08.540988 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:08.540954 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-util" (OuterVolumeSpecName: "util") pod "aefe18e7-ab0f-4eca-9ced-6714a53c3c94" (UID: "aefe18e7-ab0f-4eca-9ced-6714a53c3c94"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:27:08.634206 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:08.634177 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-util\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:27:08.634206 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:08.634202 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4j2g\" (UniqueName: \"kubernetes.io/projected/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-kube-api-access-m4j2g\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:27:08.634360 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:08.634213 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aefe18e7-ab0f-4eca-9ced-6714a53c3c94-bundle\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:27:09.265656 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:09.265613 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" event={"ID":"aefe18e7-ab0f-4eca-9ced-6714a53c3c94","Type":"ContainerDied","Data":"5eb3df377da0dd1d080c16801728b5541c7d806bb7d3bb0b73809b1589e31bbb"} Jul 10 02:27:09.265656 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:09.265640 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcf4jw" Jul 10 02:27:09.265849 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:09.265673 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eb3df377da0dd1d080c16801728b5541c7d806bb7d3bb0b73809b1589e31bbb" Jul 10 02:27:17.916457 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.916366 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d"] Jul 10 02:27:17.917016 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.916731 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aefe18e7-ab0f-4eca-9ced-6714a53c3c94" containerName="util" Jul 10 02:27:17.917016 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.916748 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aefe18e7-ab0f-4eca-9ced-6714a53c3c94" containerName="util" Jul 10 02:27:17.917016 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.916764 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aefe18e7-ab0f-4eca-9ced-6714a53c3c94" containerName="extract" Jul 10 02:27:17.917016 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.916773 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aefe18e7-ab0f-4eca-9ced-6714a53c3c94" containerName="extract" Jul 10 02:27:17.917016 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.916791 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aefe18e7-ab0f-4eca-9ced-6714a53c3c94" containerName="pull" Jul 10 02:27:17.917016 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.916800 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aefe18e7-ab0f-4eca-9ced-6714a53c3c94" containerName="pull" Jul 10 02:27:17.917016 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.916861 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="aefe18e7-ab0f-4eca-9ced-6714a53c3c94" containerName="extract" Jul 10 02:27:17.918964 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.918944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:17.921874 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.921844 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-j45p7\"" Jul 10 02:27:17.922393 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.922376 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 02:27:17.922785 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.922772 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 02:27:17.923164 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.923142 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 02:27:17.923275 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.923196 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 02:27:17.946626 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.946548 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d"] Jul 10 02:27:17.993374 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.993338 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7c894dc-5d38-4a54-8448-5bed2c99389b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:17.993540 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.993387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7c894dc-5d38-4a54-8448-5bed2c99389b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:17.993540 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.993425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7c894dc-5d38-4a54-8448-5bed2c99389b-manifests\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:17.993540 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.993496 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbs25\" (UniqueName: \"kubernetes.io/projected/d7c894dc-5d38-4a54-8448-5bed2c99389b-kube-api-access-jbs25\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:17.993684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:17.993538 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7c894dc-5d38-4a54-8448-5bed2c99389b-tmp\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:18.094108 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:18.094076 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7c894dc-5d38-4a54-8448-5bed2c99389b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:18.094288 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:18.094117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7c894dc-5d38-4a54-8448-5bed2c99389b-manifests\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:18.094288 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:18.094172 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbs25\" (UniqueName: \"kubernetes.io/projected/d7c894dc-5d38-4a54-8448-5bed2c99389b-kube-api-access-jbs25\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:18.094288 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:18.094200 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7c894dc-5d38-4a54-8448-5bed2c99389b-tmp\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:18.094288 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:18.094228 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7c894dc-5d38-4a54-8448-5bed2c99389b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:18.094638 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:18.094614 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7c894dc-5d38-4a54-8448-5bed2c99389b-manifests\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:18.096694 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:18.096665 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7c894dc-5d38-4a54-8448-5bed2c99389b-tmp\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:18.096694 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:18.096683 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7c894dc-5d38-4a54-8448-5bed2c99389b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:18.096820 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:18.096767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7c894dc-5d38-4a54-8448-5bed2c99389b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:18.115034 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:18.115008 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbs25\" (UniqueName: \"kubernetes.io/projected/d7c894dc-5d38-4a54-8448-5bed2c99389b-kube-api-access-jbs25\") pod \"opendatahub-operator-controller-manager-7fc46458b-tkz2d\" (UID: \"d7c894dc-5d38-4a54-8448-5bed2c99389b\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:18.228896 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:18.228809 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:18.388951 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:18.388921 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d"] Jul 10 02:27:18.391225 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:27:18.391204 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c894dc_5d38_4a54_8448_5bed2c99389b.slice/crio-959453c8d5176d04807c24212e4a182c345ed91e9286085c0f1aa96864553b0f WatchSource:0}: Error finding container 959453c8d5176d04807c24212e4a182c345ed91e9286085c0f1aa96864553b0f: Status 404 returned error can't find the container with id 959453c8d5176d04807c24212e4a182c345ed91e9286085c0f1aa96864553b0f Jul 10 02:27:19.313464 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:19.313422 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" event={"ID":"d7c894dc-5d38-4a54-8448-5bed2c99389b","Type":"ContainerStarted","Data":"959453c8d5176d04807c24212e4a182c345ed91e9286085c0f1aa96864553b0f"} Jul 10 02:27:22.328967 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:22.328927 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" event={"ID":"d7c894dc-5d38-4a54-8448-5bed2c99389b","Type":"ContainerStarted","Data":"d1bb75700632368aa6a8d760039064d16e32141ac90e93a0a3f330b34433ba66"} Jul 10 02:27:23.334946 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:23.334910 2575 generic.go:356] "Generic (PLEG): container finished" podID="d7c894dc-5d38-4a54-8448-5bed2c99389b" containerID="d1bb75700632368aa6a8d760039064d16e32141ac90e93a0a3f330b34433ba66" exitCode=0 Jul 10 02:27:23.335350 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:23.334993 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" event={"ID":"d7c894dc-5d38-4a54-8448-5bed2c99389b","Type":"ContainerDied","Data":"d1bb75700632368aa6a8d760039064d16e32141ac90e93a0a3f330b34433ba66"} Jul 10 02:27:24.340572 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:24.340536 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" event={"ID":"d7c894dc-5d38-4a54-8448-5bed2c99389b","Type":"ContainerStarted","Data":"23a03069c39e09fa0acac4aaf40aa3ea43fae0fa4164367e49b1296304306c7b"} Jul 10 02:27:24.340954 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:24.340800 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:24.403612 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:24.403559 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" podStartSLOduration=2.995886991 podStartE2EDuration="7.403544107s" totalImagesPullingTime="3.339081205s" totalInitContainerRuntime="1.068575911s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:27:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:27:24.400012939 +0000 UTC m=+273.799465852" watchObservedRunningTime="2026-07-10 02:27:24.403544107 +0000 UTC m=+273.802997020" Jul 10 02:27:35.347281 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:35.347226 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7fc46458b-tkz2d" Jul 10 02:27:38.768807 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.768772 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-f74c66c44-gm26z"] Jul 10 02:27:38.779460 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.779434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" Jul 10 02:27:38.782628 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.782601 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xvrm2\"" Jul 10 02:27:38.782954 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.782924 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 02:27:38.786543 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.786519 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 02:27:38.789493 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.788300 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 02:27:38.789579 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.788374 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 02:27:38.791304 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.791280 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f74c66c44-gm26z"] Jul 10 02:27:38.849311 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.849274 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/301130f0-ce08-428f-9171-f65ca4ca8d98-tls-certs\") pod \"kube-auth-proxy-f74c66c44-gm26z\" (UID: \"301130f0-ce08-428f-9171-f65ca4ca8d98\") " pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" Jul 10 02:27:38.849495 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.849333 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb28r\" (UniqueName: \"kubernetes.io/projected/301130f0-ce08-428f-9171-f65ca4ca8d98-kube-api-access-bb28r\") pod \"kube-auth-proxy-f74c66c44-gm26z\" (UID: \"301130f0-ce08-428f-9171-f65ca4ca8d98\") " pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" Jul 10 02:27:38.849495 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.849362 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/301130f0-ce08-428f-9171-f65ca4ca8d98-tmp\") pod \"kube-auth-proxy-f74c66c44-gm26z\" (UID: \"301130f0-ce08-428f-9171-f65ca4ca8d98\") " pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" Jul 10 02:27:38.935058 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.935024 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vbqbf"] Jul 10 02:27:38.937529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.937511 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-vbqbf" Jul 10 02:27:38.944541 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.944518 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 02:27:38.944644 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.944521 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-vsddr\"" Jul 10 02:27:38.950441 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.950422 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/301130f0-ce08-428f-9171-f65ca4ca8d98-tls-certs\") pod \"kube-auth-proxy-f74c66c44-gm26z\" (UID: \"301130f0-ce08-428f-9171-f65ca4ca8d98\") " pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" Jul 10 02:27:38.950536 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.950457 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb28r\" (UniqueName: \"kubernetes.io/projected/301130f0-ce08-428f-9171-f65ca4ca8d98-kube-api-access-bb28r\") pod \"kube-auth-proxy-f74c66c44-gm26z\" (UID: \"301130f0-ce08-428f-9171-f65ca4ca8d98\") " pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" Jul 10 02:27:38.950536 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.950476 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/301130f0-ce08-428f-9171-f65ca4ca8d98-tmp\") pod \"kube-auth-proxy-f74c66c44-gm26z\" (UID: \"301130f0-ce08-428f-9171-f65ca4ca8d98\") " pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" Jul 10 02:27:38.950634 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:27:38.950552 2575 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 10 02:27:38.950682 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:27:38.950633 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301130f0-ce08-428f-9171-f65ca4ca8d98-tls-certs podName:301130f0-ce08-428f-9171-f65ca4ca8d98 nodeName:}" failed. No retries permitted until 2026-07-10 02:27:39.450617343 +0000 UTC m=+288.850070257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/301130f0-ce08-428f-9171-f65ca4ca8d98-tls-certs") pod "kube-auth-proxy-f74c66c44-gm26z" (UID: "301130f0-ce08-428f-9171-f65ca4ca8d98") : secret "kube-auth-proxy-tls" not found Jul 10 02:27:38.952651 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.952633 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/301130f0-ce08-428f-9171-f65ca4ca8d98-tmp\") pod \"kube-auth-proxy-f74c66c44-gm26z\" (UID: \"301130f0-ce08-428f-9171-f65ca4ca8d98\") " pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" Jul 10 02:27:38.955040 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.955017 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vbqbf"] Jul 10 02:27:38.968185 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:38.968162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb28r\" (UniqueName: \"kubernetes.io/projected/301130f0-ce08-428f-9171-f65ca4ca8d98-kube-api-access-bb28r\") pod \"kube-auth-proxy-f74c66c44-gm26z\" (UID: \"301130f0-ce08-428f-9171-f65ca4ca8d98\") " pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" Jul 10 02:27:39.051373 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:39.051290 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cae723c-e55b-434c-b2af-b69a0566a41f-cert\") pod \"odh-model-controller-7998c67dc4-vbqbf\" (UID: \"7cae723c-e55b-434c-b2af-b69a0566a41f\") " pod="opendatahub/odh-model-controller-7998c67dc4-vbqbf" Jul 10 02:27:39.051530 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:39.051415 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5mzz\" (UniqueName: \"kubernetes.io/projected/7cae723c-e55b-434c-b2af-b69a0566a41f-kube-api-access-c5mzz\") pod \"odh-model-controller-7998c67dc4-vbqbf\" (UID: \"7cae723c-e55b-434c-b2af-b69a0566a41f\") " pod="opendatahub/odh-model-controller-7998c67dc4-vbqbf" Jul 10 02:27:39.152471 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:39.152415 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5mzz\" (UniqueName: \"kubernetes.io/projected/7cae723c-e55b-434c-b2af-b69a0566a41f-kube-api-access-c5mzz\") pod \"odh-model-controller-7998c67dc4-vbqbf\" (UID: \"7cae723c-e55b-434c-b2af-b69a0566a41f\") " pod="opendatahub/odh-model-controller-7998c67dc4-vbqbf" Jul 10 02:27:39.152661 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:39.152516 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cae723c-e55b-434c-b2af-b69a0566a41f-cert\") pod \"odh-model-controller-7998c67dc4-vbqbf\" (UID: \"7cae723c-e55b-434c-b2af-b69a0566a41f\") " pod="opendatahub/odh-model-controller-7998c67dc4-vbqbf" Jul 10 02:27:39.152661 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:27:39.152624 2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 02:27:39.152766 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:27:39.152694 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cae723c-e55b-434c-b2af-b69a0566a41f-cert podName:7cae723c-e55b-434c-b2af-b69a0566a41f nodeName:}" failed. No retries permitted until 2026-07-10 02:27:39.652674684 +0000 UTC m=+289.052127578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7cae723c-e55b-434c-b2af-b69a0566a41f-cert") pod "odh-model-controller-7998c67dc4-vbqbf" (UID: "7cae723c-e55b-434c-b2af-b69a0566a41f") : secret "odh-model-controller-webhook-cert" not found Jul 10 02:27:39.162984 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:39.162955 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5mzz\" (UniqueName: \"kubernetes.io/projected/7cae723c-e55b-434c-b2af-b69a0566a41f-kube-api-access-c5mzz\") pod \"odh-model-controller-7998c67dc4-vbqbf\" (UID: \"7cae723c-e55b-434c-b2af-b69a0566a41f\") " pod="opendatahub/odh-model-controller-7998c67dc4-vbqbf" Jul 10 02:27:39.456481 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:39.456438 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/301130f0-ce08-428f-9171-f65ca4ca8d98-tls-certs\") pod \"kube-auth-proxy-f74c66c44-gm26z\" (UID: \"301130f0-ce08-428f-9171-f65ca4ca8d98\") " pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" Jul 10 02:27:39.458872 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:39.458836 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/301130f0-ce08-428f-9171-f65ca4ca8d98-tls-certs\") pod \"kube-auth-proxy-f74c66c44-gm26z\" (UID: \"301130f0-ce08-428f-9171-f65ca4ca8d98\") " pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" Jul 10 02:27:39.658366 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:39.658323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cae723c-e55b-434c-b2af-b69a0566a41f-cert\") pod \"odh-model-controller-7998c67dc4-vbqbf\" (UID: \"7cae723c-e55b-434c-b2af-b69a0566a41f\") " pod="opendatahub/odh-model-controller-7998c67dc4-vbqbf" Jul 10 02:27:39.660771 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:39.660746 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cae723c-e55b-434c-b2af-b69a0566a41f-cert\") pod \"odh-model-controller-7998c67dc4-vbqbf\" (UID: \"7cae723c-e55b-434c-b2af-b69a0566a41f\") " pod="opendatahub/odh-model-controller-7998c67dc4-vbqbf" Jul 10 02:27:39.694611 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:39.694569 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" Jul 10 02:27:39.847183 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:39.847145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-vbqbf" Jul 10 02:27:39.849769 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:39.849745 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f74c66c44-gm26z"] Jul 10 02:27:39.852489 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:27:39.852454 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301130f0_ce08_428f_9171_f65ca4ca8d98.slice/crio-93cc12877c26f7c9705798b36b4357d0dc8897136308c0b953b4cdf65d709a84 WatchSource:0}: Error finding container 93cc12877c26f7c9705798b36b4357d0dc8897136308c0b953b4cdf65d709a84: Status 404 returned error can't find the container with id 93cc12877c26f7c9705798b36b4357d0dc8897136308c0b953b4cdf65d709a84 Jul 10 02:27:40.000387 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.000363 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vbqbf"] Jul 10 02:27:40.072764 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.072722 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kxtp7"] Jul 10 02:27:40.076189 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.076170 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-kxtp7" Jul 10 02:27:40.079275 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.079255 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 02:27:40.087291 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.087200 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kxtp7"] Jul 10 02:27:40.161089 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.161053 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km5gr\" (UniqueName: \"kubernetes.io/projected/bb190fb4-b0f8-409b-9c9c-194dc761339b-kube-api-access-km5gr\") pod \"model-serving-api-54d89bd5fc-kxtp7\" (UID: \"bb190fb4-b0f8-409b-9c9c-194dc761339b\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxtp7" Jul 10 02:27:40.161294 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.161226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb190fb4-b0f8-409b-9c9c-194dc761339b-tls-certs\") pod \"model-serving-api-54d89bd5fc-kxtp7\" (UID: \"bb190fb4-b0f8-409b-9c9c-194dc761339b\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxtp7" Jul 10 02:27:40.262467 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.262365 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km5gr\" (UniqueName: \"kubernetes.io/projected/bb190fb4-b0f8-409b-9c9c-194dc761339b-kube-api-access-km5gr\") pod \"model-serving-api-54d89bd5fc-kxtp7\" (UID: \"bb190fb4-b0f8-409b-9c9c-194dc761339b\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxtp7" Jul 10 02:27:40.262467 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.262419 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb190fb4-b0f8-409b-9c9c-194dc761339b-tls-certs\") pod \"model-serving-api-54d89bd5fc-kxtp7\" (UID: \"bb190fb4-b0f8-409b-9c9c-194dc761339b\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxtp7" Jul 10 02:27:40.265040 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.265011 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb190fb4-b0f8-409b-9c9c-194dc761339b-tls-certs\") pod \"model-serving-api-54d89bd5fc-kxtp7\" (UID: \"bb190fb4-b0f8-409b-9c9c-194dc761339b\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxtp7" Jul 10 02:27:40.281439 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.281404 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km5gr\" (UniqueName: \"kubernetes.io/projected/bb190fb4-b0f8-409b-9c9c-194dc761339b-kube-api-access-km5gr\") pod \"model-serving-api-54d89bd5fc-kxtp7\" (UID: \"bb190fb4-b0f8-409b-9c9c-194dc761339b\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxtp7" Jul 10 02:27:40.379473 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.379444 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s"] Jul 10 02:27:40.382088 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.382069 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.386751 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.386500 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 02:27:40.386751 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.386504 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 02:27:40.386751 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.386510 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 02:27:40.386751 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.386666 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c6md5\"" Jul 10 02:27:40.387036 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.386816 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 02:27:40.389766 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.389552 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-kxtp7" Jul 10 02:27:40.404043 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.404016 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s"] Jul 10 02:27:40.409791 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.409762 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vbqbf" event={"ID":"7cae723c-e55b-434c-b2af-b69a0566a41f","Type":"ContainerStarted","Data":"f45db75da0ae2173fa99c410d4b4415ec5946e50c41b6ff9d60d417e7627419c"} Jul 10 02:27:40.410970 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.410941 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" event={"ID":"301130f0-ce08-428f-9171-f65ca4ca8d98","Type":"ContainerStarted","Data":"93cc12877c26f7c9705798b36b4357d0dc8897136308c0b953b4cdf65d709a84"} Jul 10 02:27:40.463869 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.463832 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.464033 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.463896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.464033 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.463964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4q7b\" (UniqueName: \"kubernetes.io/projected/25f96dbf-0b71-487d-8eb5-3eab146a6c22-kube-api-access-l4q7b\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.464033 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.463993 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-cacerts\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.464033 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.464024 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.464237 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.464067 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f96dbf-0b71-487d-8eb5-3eab146a6c22-local-certs\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.464237 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.464161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-token\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.542396 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.542364 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kxtp7"] Jul 10 02:27:40.544635 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:27:40.544602 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb190fb4_b0f8_409b_9c9c_194dc761339b.slice/crio-ba7ea5073bd0229a7a1307de3934af1e5aa247bd8c5746241bb63b5d993c38cd WatchSource:0}: Error finding container ba7ea5073bd0229a7a1307de3934af1e5aa247bd8c5746241bb63b5d993c38cd: Status 404 returned error can't find the container with id ba7ea5073bd0229a7a1307de3934af1e5aa247bd8c5746241bb63b5d993c38cd Jul 10 02:27:40.564975 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.564947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.565114 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.565018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4q7b\" (UniqueName: \"kubernetes.io/projected/25f96dbf-0b71-487d-8eb5-3eab146a6c22-kube-api-access-l4q7b\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.565114 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.565051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-cacerts\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.565114 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.565086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.565312 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.565220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f96dbf-0b71-487d-8eb5-3eab146a6c22-local-certs\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.565312 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.565305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-token\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.565413 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.565347 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.565725 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.565666 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.567909 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.567863 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f96dbf-0b71-487d-8eb5-3eab146a6c22-local-certs\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.568036 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.568010 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.568199 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.568176 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-cacerts\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.568739 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.568717 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.579054 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.579011 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4q7b\" (UniqueName: \"kubernetes.io/projected/25f96dbf-0b71-487d-8eb5-3eab146a6c22-kube-api-access-l4q7b\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.580194 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.580172 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-token\") pod \"istiod-openshift-gateway-7574459656-kpv6s\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.692769 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.692443 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:40.879890 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:40.879795 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s"] Jul 10 02:27:40.886962 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:27:40.886926 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f96dbf_0b71_487d_8eb5_3eab146a6c22.slice/crio-c847b008d50cf5811115daf32040200564a672140ea005c2918e309ae00aa290 WatchSource:0}: Error finding container c847b008d50cf5811115daf32040200564a672140ea005c2918e309ae00aa290: Status 404 returned error can't find the container with id c847b008d50cf5811115daf32040200564a672140ea005c2918e309ae00aa290 Jul 10 02:27:41.429475 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:41.424704 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" event={"ID":"25f96dbf-0b71-487d-8eb5-3eab146a6c22","Type":"ContainerStarted","Data":"c847b008d50cf5811115daf32040200564a672140ea005c2918e309ae00aa290"} Jul 10 02:27:41.431209 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:41.431176 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-kxtp7" event={"ID":"bb190fb4-b0f8-409b-9c9c-194dc761339b","Type":"ContainerStarted","Data":"ba7ea5073bd0229a7a1307de3934af1e5aa247bd8c5746241bb63b5d993c38cd"} Jul 10 02:27:44.447379 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:44.447280 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vbqbf" event={"ID":"7cae723c-e55b-434c-b2af-b69a0566a41f","Type":"ContainerStarted","Data":"2011f93ea9575d29ceb398c9110907328a84edf36c121f6f6c723a498bfed2ce"} Jul 10 02:27:44.447797 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:44.447496 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-vbqbf" Jul 10 02:27:44.448946 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:44.448919 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" event={"ID":"301130f0-ce08-428f-9171-f65ca4ca8d98","Type":"ContainerStarted","Data":"1a142714b158cbbc45cd012cc496da6140144588966887b352da502acbf1b277"} Jul 10 02:27:44.450496 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:44.450477 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-kxtp7" event={"ID":"bb190fb4-b0f8-409b-9c9c-194dc761339b","Type":"ContainerStarted","Data":"6c3c71298e0886497f0cce0ec060f7d5a8558400b07ddea592cd9a2b10aa9093"} Jul 10 02:27:44.450610 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:44.450595 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-kxtp7" Jul 10 02:27:44.467289 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:44.467224 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-vbqbf" podStartSLOduration=2.279630663 podStartE2EDuration="6.467206383s" totalImagesPullingTime="4.18757572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:27:44.465557579 +0000 UTC m=+293.865010492" watchObservedRunningTime="2026-07-10 02:27:44.467206383 +0000 UTC m=+293.866659279" Jul 10 02:27:44.487290 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:44.487225 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-kxtp7" podStartSLOduration=0.833987184 podStartE2EDuration="4.48720609s" totalImagesPullingTime="3.653218906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:27:44.483144225 +0000 UTC m=+293.882597142" watchObservedRunningTime="2026-07-10 02:27:44.48720609 +0000 UTC m=+293.886659008" Jul 10 02:27:44.505035 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:44.504906 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-f74c66c44-gm26z" podStartSLOduration=2.160022382 podStartE2EDuration="6.504885834s" totalImagesPullingTime="4.344863452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:27:44.502747093 +0000 UTC m=+293.902200006" watchObservedRunningTime="2026-07-10 02:27:44.504885834 +0000 UTC m=+293.904338750" Jul 10 02:27:45.746481 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:45.746439 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 02:27:45.746745 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:45.746510 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 02:27:46.462374 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:46.462340 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" event={"ID":"25f96dbf-0b71-487d-8eb5-3eab146a6c22","Type":"ContainerStarted","Data":"9ddaf8228de8021903e2149630a44b53f23bbbc297b059c5d34c0e69b1b0649f"} Jul 10 02:27:46.462615 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:46.462588 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:46.464636 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:46.464603 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-kpv6s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 02:27:46.464772 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:46.464679 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" podUID="25f96dbf-0b71-487d-8eb5-3eab146a6c22" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 02:27:47.469315 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:47.469291 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:47.564866 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:47.564816 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" podStartSLOduration=2.708259429 podStartE2EDuration="7.564796715s" totalImagesPullingTime="4.856537286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:27:46.489202367 +0000 UTC m=+295.888655280" watchObservedRunningTime="2026-07-10 02:27:47.564796715 +0000 UTC m=+296.964249629" Jul 10 02:27:51.016285 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.016238 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:27:51.017749 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.017726 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:27:51.021911 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.021893 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 02:27:51.350601 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.350420 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw"] Jul 10 02:27:51.354426 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.354409 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw" Jul 10 02:27:51.357646 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.357623 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 10 02:27:51.357646 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.357634 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mgbkm\"" Jul 10 02:27:51.371748 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.371727 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw"] Jul 10 02:27:51.401533 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.401504 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z"] Jul 10 02:27:51.403912 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.403879 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z" Jul 10 02:27:51.412294 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.412273 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 10 02:27:51.412661 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.412646 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-zz98h\"" Jul 10 02:27:51.429707 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.429681 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z"] Jul 10 02:27:51.463160 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.463136 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhgq\" (UniqueName: \"kubernetes.io/projected/77b501f6-3b1b-4da6-8e58-d3c5c99edcbf-kube-api-access-tqhgq\") pod \"llmisvc-controller-manager-777c6bd7d5-ps52z\" (UID: \"77b501f6-3b1b-4da6-8e58-d3c5c99edcbf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z" Jul 10 02:27:51.463305 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.463179 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8838bec-0054-46b7-9fc0-9bf5089f0949-cert\") pod \"kserve-controller-manager-d6f4b65b8-kjvkw\" (UID: \"b8838bec-0054-46b7-9fc0-9bf5089f0949\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw" Jul 10 02:27:51.463305 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.463202 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6qct\" (UniqueName: \"kubernetes.io/projected/b8838bec-0054-46b7-9fc0-9bf5089f0949-kube-api-access-z6qct\") pod \"kserve-controller-manager-d6f4b65b8-kjvkw\" (UID: \"b8838bec-0054-46b7-9fc0-9bf5089f0949\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw" Jul 10 02:27:51.463381 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.463304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77b501f6-3b1b-4da6-8e58-d3c5c99edcbf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ps52z\" (UID: \"77b501f6-3b1b-4da6-8e58-d3c5c99edcbf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z" Jul 10 02:27:51.564727 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.564691 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhgq\" (UniqueName: \"kubernetes.io/projected/77b501f6-3b1b-4da6-8e58-d3c5c99edcbf-kube-api-access-tqhgq\") pod \"llmisvc-controller-manager-777c6bd7d5-ps52z\" (UID: \"77b501f6-3b1b-4da6-8e58-d3c5c99edcbf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z" Jul 10 02:27:51.564888 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.564739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8838bec-0054-46b7-9fc0-9bf5089f0949-cert\") pod \"kserve-controller-manager-d6f4b65b8-kjvkw\" (UID: \"b8838bec-0054-46b7-9fc0-9bf5089f0949\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw" Jul 10 02:27:51.564888 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.564767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qct\" (UniqueName: \"kubernetes.io/projected/b8838bec-0054-46b7-9fc0-9bf5089f0949-kube-api-access-z6qct\") pod \"kserve-controller-manager-d6f4b65b8-kjvkw\" (UID: \"b8838bec-0054-46b7-9fc0-9bf5089f0949\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw" Jul 10 02:27:51.564888 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.564800 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77b501f6-3b1b-4da6-8e58-d3c5c99edcbf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ps52z\" (UID: \"77b501f6-3b1b-4da6-8e58-d3c5c99edcbf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z" Jul 10 02:27:51.567296 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.567271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77b501f6-3b1b-4da6-8e58-d3c5c99edcbf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ps52z\" (UID: \"77b501f6-3b1b-4da6-8e58-d3c5c99edcbf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z" Jul 10 02:27:51.567399 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.567338 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8838bec-0054-46b7-9fc0-9bf5089f0949-cert\") pod \"kserve-controller-manager-d6f4b65b8-kjvkw\" (UID: \"b8838bec-0054-46b7-9fc0-9bf5089f0949\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw" Jul 10 02:27:51.574558 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.574533 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhgq\" (UniqueName: \"kubernetes.io/projected/77b501f6-3b1b-4da6-8e58-d3c5c99edcbf-kube-api-access-tqhgq\") pod \"llmisvc-controller-manager-777c6bd7d5-ps52z\" (UID: \"77b501f6-3b1b-4da6-8e58-d3c5c99edcbf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z" Jul 10 02:27:51.575516 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.575501 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qct\" (UniqueName: \"kubernetes.io/projected/b8838bec-0054-46b7-9fc0-9bf5089f0949-kube-api-access-z6qct\") pod \"kserve-controller-manager-d6f4b65b8-kjvkw\" (UID: \"b8838bec-0054-46b7-9fc0-9bf5089f0949\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw" Jul 10 02:27:51.665126 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.665090 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw" Jul 10 02:27:51.713283 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.713238 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z" Jul 10 02:27:51.855555 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.854557 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw"] Jul 10 02:27:51.857354 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:27:51.857328 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8838bec_0054_46b7_9fc0_9bf5089f0949.slice/crio-36fdada64843f6dca653657a8d3fd4d3ee6c5c9c1ccac96a75f73f5219d9bf92 WatchSource:0}: Error finding container 36fdada64843f6dca653657a8d3fd4d3ee6c5c9c1ccac96a75f73f5219d9bf92: Status 404 returned error can't find the container with id 36fdada64843f6dca653657a8d3fd4d3ee6c5c9c1ccac96a75f73f5219d9bf92 Jul 10 02:27:51.884611 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:51.884570 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z"] Jul 10 02:27:51.886924 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:27:51.886900 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod77b501f6_3b1b_4da6_8e58_d3c5c99edcbf.slice/crio-ef726fc228901b0a0f9d89885fac3469c98c9d6df37cad159cc88b9cd787083c WatchSource:0}: Error finding container ef726fc228901b0a0f9d89885fac3469c98c9d6df37cad159cc88b9cd787083c: Status 404 returned error can't find the container with id ef726fc228901b0a0f9d89885fac3469c98c9d6df37cad159cc88b9cd787083c Jul 10 02:27:52.493016 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:52.492974 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z" event={"ID":"77b501f6-3b1b-4da6-8e58-d3c5c99edcbf","Type":"ContainerStarted","Data":"ef726fc228901b0a0f9d89885fac3469c98c9d6df37cad159cc88b9cd787083c"} Jul 10 02:27:52.494014 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:52.493993 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw" event={"ID":"b8838bec-0054-46b7-9fc0-9bf5089f0949","Type":"ContainerStarted","Data":"36fdada64843f6dca653657a8d3fd4d3ee6c5c9c1ccac96a75f73f5219d9bf92"} Jul 10 02:27:53.575967 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.575931 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s"] Jul 10 02:27:53.576430 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.576160 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" podUID="25f96dbf-0b71-487d-8eb5-3eab146a6c22" containerName="discovery" containerID="cri-o://9ddaf8228de8021903e2149630a44b53f23bbbc297b059c5d34c0e69b1b0649f" gracePeriod=30 Jul 10 02:27:53.832497 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.832439 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:53.885170 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.885136 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-cacerts\") pod \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " Jul 10 02:27:53.885342 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.885190 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-kubeconfig\") pod \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " Jul 10 02:27:53.885342 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.885237 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4q7b\" (UniqueName: \"kubernetes.io/projected/25f96dbf-0b71-487d-8eb5-3eab146a6c22-kube-api-access-l4q7b\") pod \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " Jul 10 02:27:53.885342 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.885283 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f96dbf-0b71-487d-8eb5-3eab146a6c22-local-certs\") pod \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " Jul 10 02:27:53.885342 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.885317 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-csr-dns-cert\") pod \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " Jul 10 02:27:53.885342 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.885335 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-csr-ca-configmap\") pod \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " Jul 10 02:27:53.885615 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.885360 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-token\") pod \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\" (UID: \"25f96dbf-0b71-487d-8eb5-3eab146a6c22\") " Jul 10 02:27:53.886060 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.886029 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "25f96dbf-0b71-487d-8eb5-3eab146a6c22" (UID: "25f96dbf-0b71-487d-8eb5-3eab146a6c22"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:27:53.888495 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.888454 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "25f96dbf-0b71-487d-8eb5-3eab146a6c22" (UID: "25f96dbf-0b71-487d-8eb5-3eab146a6c22"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:27:53.888659 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.888628 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-token" (OuterVolumeSpecName: "istio-token") pod "25f96dbf-0b71-487d-8eb5-3eab146a6c22" (UID: "25f96dbf-0b71-487d-8eb5-3eab146a6c22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:27:53.888659 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.888648 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f96dbf-0b71-487d-8eb5-3eab146a6c22-local-certs" (OuterVolumeSpecName: "local-certs") pod "25f96dbf-0b71-487d-8eb5-3eab146a6c22" (UID: "25f96dbf-0b71-487d-8eb5-3eab146a6c22"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:27:53.888813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.888638 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "25f96dbf-0b71-487d-8eb5-3eab146a6c22" (UID: "25f96dbf-0b71-487d-8eb5-3eab146a6c22"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:27:53.888813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.888684 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-cacerts" (OuterVolumeSpecName: "cacerts") pod "25f96dbf-0b71-487d-8eb5-3eab146a6c22" (UID: "25f96dbf-0b71-487d-8eb5-3eab146a6c22"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:27:53.889775 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.889753 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f96dbf-0b71-487d-8eb5-3eab146a6c22-kube-api-access-l4q7b" (OuterVolumeSpecName: "kube-api-access-l4q7b") pod "25f96dbf-0b71-487d-8eb5-3eab146a6c22" (UID: "25f96dbf-0b71-487d-8eb5-3eab146a6c22"). InnerVolumeSpecName "kube-api-access-l4q7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:27:53.986690 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.986650 2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-cacerts\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:27:53.986690 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.986681 2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-kubeconfig\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:27:53.986690 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.986693 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4q7b\" (UniqueName: \"kubernetes.io/projected/25f96dbf-0b71-487d-8eb5-3eab146a6c22-kube-api-access-l4q7b\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:27:53.986993 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.986706 2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f96dbf-0b71-487d-8eb5-3eab146a6c22-local-certs\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:27:53.986993 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.986720 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-csr-dns-cert\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:27:53.986993 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.986732 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-csr-ca-configmap\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:27:53.986993 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:53.986747 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f96dbf-0b71-487d-8eb5-3eab146a6c22-istio-token\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:27:54.508965 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:54.508874 2575 generic.go:356] "Generic (PLEG): container finished" podID="25f96dbf-0b71-487d-8eb5-3eab146a6c22" containerID="9ddaf8228de8021903e2149630a44b53f23bbbc297b059c5d34c0e69b1b0649f" exitCode=0 Jul 10 02:27:54.509132 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:54.508979 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" Jul 10 02:27:54.509132 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:54.508994 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" event={"ID":"25f96dbf-0b71-487d-8eb5-3eab146a6c22","Type":"ContainerDied","Data":"9ddaf8228de8021903e2149630a44b53f23bbbc297b059c5d34c0e69b1b0649f"} Jul 10 02:27:54.509132 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:54.509048 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s" event={"ID":"25f96dbf-0b71-487d-8eb5-3eab146a6c22","Type":"ContainerDied","Data":"c847b008d50cf5811115daf32040200564a672140ea005c2918e309ae00aa290"} Jul 10 02:27:54.509132 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:54.509076 2575 scope.go:117] "RemoveContainer" containerID="9ddaf8228de8021903e2149630a44b53f23bbbc297b059c5d34c0e69b1b0649f" Jul 10 02:27:54.537682 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:54.537657 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s"] Jul 10 02:27:54.541943 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:54.541915 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kpv6s"] Jul 10 02:27:55.129045 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:55.129013 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25f96dbf-0b71-487d-8eb5-3eab146a6c22" path="/var/lib/kubelet/pods/25f96dbf-0b71-487d-8eb5-3eab146a6c22/volumes" Jul 10 02:27:55.186772 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:55.186592 2575 scope.go:117] "RemoveContainer" containerID="9ddaf8228de8021903e2149630a44b53f23bbbc297b059c5d34c0e69b1b0649f" Jul 10 02:27:55.186918 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:27:55.186897 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ddaf8228de8021903e2149630a44b53f23bbbc297b059c5d34c0e69b1b0649f\": container with ID starting with 9ddaf8228de8021903e2149630a44b53f23bbbc297b059c5d34c0e69b1b0649f not found: ID does not exist" containerID="9ddaf8228de8021903e2149630a44b53f23bbbc297b059c5d34c0e69b1b0649f" Jul 10 02:27:55.186967 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:55.186928 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ddaf8228de8021903e2149630a44b53f23bbbc297b059c5d34c0e69b1b0649f"} err="failed to get container status \"9ddaf8228de8021903e2149630a44b53f23bbbc297b059c5d34c0e69b1b0649f\": rpc error: code = NotFound desc = could not find container \"9ddaf8228de8021903e2149630a44b53f23bbbc297b059c5d34c0e69b1b0649f\": container with ID starting with 9ddaf8228de8021903e2149630a44b53f23bbbc297b059c5d34c0e69b1b0649f not found: ID does not exist" Jul 10 02:27:55.459623 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:55.459537 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-vbqbf" Jul 10 02:27:55.461514 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:55.461493 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-kxtp7" Jul 10 02:27:55.515056 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:55.515023 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z" event={"ID":"77b501f6-3b1b-4da6-8e58-d3c5c99edcbf","Type":"ContainerStarted","Data":"96eb024832d75577334a1245f435843e5afb4b6289a7eb98715d5d4db17c1b0d"} Jul 10 02:27:55.515216 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:55.515091 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z" Jul 10 02:27:55.516736 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:55.516713 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw" event={"ID":"b8838bec-0054-46b7-9fc0-9bf5089f0949","Type":"ContainerStarted","Data":"d040e83f50c1f2b24d049c45de9f94659b25f0997daa6da5a34d1c30b8885045"} Jul 10 02:27:55.516852 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:55.516757 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw" Jul 10 02:27:55.554398 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:55.554339 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z" podStartSLOduration=1.242279657 podStartE2EDuration="4.554318866s" totalImagesPullingTime="3.312039209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:27:55.553669628 +0000 UTC m=+304.953122543" watchObservedRunningTime="2026-07-10 02:27:55.554318866 +0000 UTC m=+304.953771780" Jul 10 02:27:55.582771 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:27:55.582721 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw" podStartSLOduration=1.245566835 podStartE2EDuration="4.582705616s" totalImagesPullingTime="3.337138781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:27:55.579719594 +0000 UTC m=+304.979172506" watchObservedRunningTime="2026-07-10 02:27:55.582705616 +0000 UTC m=+304.982158529" Jul 10 02:28:26.524049 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:26.524017 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ps52z" Jul 10 02:28:26.527059 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:26.527042 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kjvkw" Jul 10 02:28:41.044219 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.044186 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k"] Jul 10 02:28:41.044599 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.044540 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25f96dbf-0b71-487d-8eb5-3eab146a6c22" containerName="discovery" Jul 10 02:28:41.044599 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.044554 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="25f96dbf-0b71-487d-8eb5-3eab146a6c22" containerName="discovery" Jul 10 02:28:41.044667 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.044626 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="25f96dbf-0b71-487d-8eb5-3eab146a6c22" containerName="discovery" Jul 10 02:28:41.053492 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.053468 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" Jul 10 02:28:41.057805 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.057779 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 02:28:41.057990 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.057783 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-phmc2\"" Jul 10 02:28:41.058078 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.057857 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 02:28:41.058772 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.058751 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k"] Jul 10 02:28:41.134618 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.134577 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw"] Jul 10 02:28:41.136995 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.136976 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" Jul 10 02:28:41.148524 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.148499 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw"] Jul 10 02:28:41.163723 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.163689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e059e15a-f735-44cd-8584-b6e4fb85a7ee-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k\" (UID: \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" Jul 10 02:28:41.163840 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.163744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e059e15a-f735-44cd-8584-b6e4fb85a7ee-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k\" (UID: \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" Jul 10 02:28:41.163840 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.163803 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92pkg\" (UniqueName: \"kubernetes.io/projected/e059e15a-f735-44cd-8584-b6e4fb85a7ee-kube-api-access-92pkg\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k\" (UID: \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" Jul 10 02:28:41.249160 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.249132 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6"] Jul 10 02:28:41.251482 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.251467 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" Jul 10 02:28:41.264014 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.263992 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6"] Jul 10 02:28:41.264285 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.264265 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92pkg\" (UniqueName: \"kubernetes.io/projected/e059e15a-f735-44cd-8584-b6e4fb85a7ee-kube-api-access-92pkg\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k\" (UID: \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" Jul 10 02:28:41.264375 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.264313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/505dfb70-10af-47a4-9a81-46eb36e31291-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw\" (UID: \"505dfb70-10af-47a4-9a81-46eb36e31291\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" Jul 10 02:28:41.264375 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.264356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nbqb\" (UniqueName: \"kubernetes.io/projected/505dfb70-10af-47a4-9a81-46eb36e31291-kube-api-access-7nbqb\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw\" (UID: \"505dfb70-10af-47a4-9a81-46eb36e31291\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" Jul 10 02:28:41.264480 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.264391 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/505dfb70-10af-47a4-9a81-46eb36e31291-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw\" (UID: \"505dfb70-10af-47a4-9a81-46eb36e31291\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" Jul 10 02:28:41.264480 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.264435 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e059e15a-f735-44cd-8584-b6e4fb85a7ee-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k\" (UID: \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" Jul 10 02:28:41.264584 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.264505 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e059e15a-f735-44cd-8584-b6e4fb85a7ee-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k\" (UID: \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" Jul 10 02:28:41.264772 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.264754 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e059e15a-f735-44cd-8584-b6e4fb85a7ee-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k\" (UID: \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" Jul 10 02:28:41.264837 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.264818 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e059e15a-f735-44cd-8584-b6e4fb85a7ee-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k\" (UID: \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" Jul 10 02:28:41.276197 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.276176 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92pkg\" (UniqueName: \"kubernetes.io/projected/e059e15a-f735-44cd-8584-b6e4fb85a7ee-kube-api-access-92pkg\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k\" (UID: \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" Jul 10 02:28:41.351370 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.351301 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46"] Jul 10 02:28:41.353871 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.353856 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" Jul 10 02:28:41.363731 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.363708 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" Jul 10 02:28:41.364399 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.364380 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46"] Jul 10 02:28:41.364977 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.364957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6\" (UID: \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" Jul 10 02:28:41.365025 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.364994 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv7pg\" (UniqueName: \"kubernetes.io/projected/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-kube-api-access-kv7pg\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6\" (UID: \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" Jul 10 02:28:41.365064 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.365024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/505dfb70-10af-47a4-9a81-46eb36e31291-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw\" (UID: \"505dfb70-10af-47a4-9a81-46eb36e31291\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" Jul 10 02:28:41.365240 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.365222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nbqb\" (UniqueName: \"kubernetes.io/projected/505dfb70-10af-47a4-9a81-46eb36e31291-kube-api-access-7nbqb\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw\" (UID: \"505dfb70-10af-47a4-9a81-46eb36e31291\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" Jul 10 02:28:41.365319 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.365272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/505dfb70-10af-47a4-9a81-46eb36e31291-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw\" (UID: \"505dfb70-10af-47a4-9a81-46eb36e31291\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" Jul 10 02:28:41.365377 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.365325 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/505dfb70-10af-47a4-9a81-46eb36e31291-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw\" (UID: \"505dfb70-10af-47a4-9a81-46eb36e31291\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" Jul 10 02:28:41.365377 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.365338 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6\" (UID: \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" Jul 10 02:28:41.365619 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.365596 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/505dfb70-10af-47a4-9a81-46eb36e31291-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw\" (UID: \"505dfb70-10af-47a4-9a81-46eb36e31291\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" Jul 10 02:28:41.377233 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.377198 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nbqb\" (UniqueName: \"kubernetes.io/projected/505dfb70-10af-47a4-9a81-46eb36e31291-kube-api-access-7nbqb\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw\" (UID: \"505dfb70-10af-47a4-9a81-46eb36e31291\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" Jul 10 02:28:41.447671 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.447637 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" Jul 10 02:28:41.466183 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.466148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5ff8126-0aee-4978-980a-f2ead93aef87-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46\" (UID: \"f5ff8126-0aee-4978-980a-f2ead93aef87\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" Jul 10 02:28:41.466336 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.466203 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxwrh\" (UniqueName: \"kubernetes.io/projected/f5ff8126-0aee-4978-980a-f2ead93aef87-kube-api-access-lxwrh\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46\" (UID: \"f5ff8126-0aee-4978-980a-f2ead93aef87\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" Jul 10 02:28:41.466336 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.466294 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5ff8126-0aee-4978-980a-f2ead93aef87-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46\" (UID: \"f5ff8126-0aee-4978-980a-f2ead93aef87\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" Jul 10 02:28:41.466455 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.466406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6\" (UID: \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" Jul 10 02:28:41.466455 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.466445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6\" (UID: \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" Jul 10 02:28:41.466565 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.466490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv7pg\" (UniqueName: \"kubernetes.io/projected/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-kube-api-access-kv7pg\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6\" (UID: \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" Jul 10 02:28:41.467563 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.467236 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6\" (UID: \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" Jul 10 02:28:41.467563 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.467505 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6\" (UID: \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" Jul 10 02:28:41.477367 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.477325 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv7pg\" (UniqueName: \"kubernetes.io/projected/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-kube-api-access-kv7pg\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6\" (UID: \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" Jul 10 02:28:41.501835 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.501791 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k"] Jul 10 02:28:41.504954 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:28:41.504920 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode059e15a_f735_44cd_8584_b6e4fb85a7ee.slice/crio-4c0307d10d2c259bf51f86e49dd4912ec11ac31e7317451d7a210339e8574474 WatchSource:0}: Error finding container 4c0307d10d2c259bf51f86e49dd4912ec11ac31e7317451d7a210339e8574474: Status 404 returned error can't find the container with id 4c0307d10d2c259bf51f86e49dd4912ec11ac31e7317451d7a210339e8574474 Jul 10 02:28:41.560441 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.560415 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" Jul 10 02:28:41.567356 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.567328 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5ff8126-0aee-4978-980a-f2ead93aef87-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46\" (UID: \"f5ff8126-0aee-4978-980a-f2ead93aef87\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" Jul 10 02:28:41.567485 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.567373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxwrh\" (UniqueName: \"kubernetes.io/projected/f5ff8126-0aee-4978-980a-f2ead93aef87-kube-api-access-lxwrh\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46\" (UID: \"f5ff8126-0aee-4978-980a-f2ead93aef87\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" Jul 10 02:28:41.567485 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.567403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5ff8126-0aee-4978-980a-f2ead93aef87-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46\" (UID: \"f5ff8126-0aee-4978-980a-f2ead93aef87\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" Jul 10 02:28:41.567935 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.567820 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5ff8126-0aee-4978-980a-f2ead93aef87-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46\" (UID: \"f5ff8126-0aee-4978-980a-f2ead93aef87\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" Jul 10 02:28:41.568159 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.568129 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5ff8126-0aee-4978-980a-f2ead93aef87-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46\" (UID: \"f5ff8126-0aee-4978-980a-f2ead93aef87\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" Jul 10 02:28:41.577691 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.577670 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxwrh\" (UniqueName: \"kubernetes.io/projected/f5ff8126-0aee-4978-980a-f2ead93aef87-kube-api-access-lxwrh\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46\" (UID: \"f5ff8126-0aee-4978-980a-f2ead93aef87\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" Jul 10 02:28:41.591553 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.591345 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw"] Jul 10 02:28:41.594859 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:28:41.594831 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505dfb70_10af_47a4_9a81_46eb36e31291.slice/crio-01d12b73ee82aab7b12f4cc76aa2b6436572306a69f2805068bac1b44838ec89 WatchSource:0}: Error finding container 01d12b73ee82aab7b12f4cc76aa2b6436572306a69f2805068bac1b44838ec89: Status 404 returned error can't find the container with id 01d12b73ee82aab7b12f4cc76aa2b6436572306a69f2805068bac1b44838ec89 Jul 10 02:28:41.662525 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.662496 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" Jul 10 02:28:41.701806 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.701703 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6"] Jul 10 02:28:41.705187 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:28:41.703932 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a2a0ec_b85e_4680_9d27_1dc6cc718f80.slice/crio-7683ffa2afce5a9489e883aacc815cb6b039475f75408fe5829748f605a6cc4d WatchSource:0}: Error finding container 7683ffa2afce5a9489e883aacc815cb6b039475f75408fe5829748f605a6cc4d: Status 404 returned error can't find the container with id 7683ffa2afce5a9489e883aacc815cb6b039475f75408fe5829748f605a6cc4d Jul 10 02:28:41.724835 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.721319 2575 generic.go:356] "Generic (PLEG): container finished" podID="e059e15a-f735-44cd-8584-b6e4fb85a7ee" containerID="a8024ca59c8f4889913a1f8a93627d20bceb0be3725ded3f1ae089ac13e51b7c" exitCode=0 Jul 10 02:28:41.724835 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.721458 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" event={"ID":"e059e15a-f735-44cd-8584-b6e4fb85a7ee","Type":"ContainerDied","Data":"a8024ca59c8f4889913a1f8a93627d20bceb0be3725ded3f1ae089ac13e51b7c"} Jul 10 02:28:41.724835 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.721507 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" event={"ID":"e059e15a-f735-44cd-8584-b6e4fb85a7ee","Type":"ContainerStarted","Data":"4c0307d10d2c259bf51f86e49dd4912ec11ac31e7317451d7a210339e8574474"} Jul 10 02:28:41.724835 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.724593 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 02:28:41.729384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.726606 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" event={"ID":"61a2a0ec-b85e-4680-9d27-1dc6cc718f80","Type":"ContainerStarted","Data":"7683ffa2afce5a9489e883aacc815cb6b039475f75408fe5829748f605a6cc4d"} Jul 10 02:28:41.730721 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.730077 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" event={"ID":"505dfb70-10af-47a4-9a81-46eb36e31291","Type":"ContainerStarted","Data":"ce854d1dc98df1efcc97a7b6709d094fdb240759180c4fe753172fea2d1848ab"} Jul 10 02:28:41.730721 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.730130 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" event={"ID":"505dfb70-10af-47a4-9a81-46eb36e31291","Type":"ContainerStarted","Data":"01d12b73ee82aab7b12f4cc76aa2b6436572306a69f2805068bac1b44838ec89"} Jul 10 02:28:41.809344 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:41.809319 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46"] Jul 10 02:28:41.812531 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:28:41.812500 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ff8126_0aee_4978_980a_f2ead93aef87.slice/crio-8ab72374f8653a3a4059e1442a58138ad5b49f42da19d016ef69dfad3018dc46 WatchSource:0}: Error finding container 8ab72374f8653a3a4059e1442a58138ad5b49f42da19d016ef69dfad3018dc46: Status 404 returned error can't find the container with id 8ab72374f8653a3a4059e1442a58138ad5b49f42da19d016ef69dfad3018dc46 Jul 10 02:28:42.736304 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:42.736269 2575 generic.go:356] "Generic (PLEG): container finished" podID="61a2a0ec-b85e-4680-9d27-1dc6cc718f80" containerID="e9220f79255e41b8345492ff1f79d502aab561446aaedee7a0a46164da5b354f" exitCode=0 Jul 10 02:28:42.736704 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:42.736341 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" event={"ID":"61a2a0ec-b85e-4680-9d27-1dc6cc718f80","Type":"ContainerDied","Data":"e9220f79255e41b8345492ff1f79d502aab561446aaedee7a0a46164da5b354f"} Jul 10 02:28:42.737930 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:42.737913 2575 generic.go:356] "Generic (PLEG): container finished" podID="505dfb70-10af-47a4-9a81-46eb36e31291" containerID="ce854d1dc98df1efcc97a7b6709d094fdb240759180c4fe753172fea2d1848ab" exitCode=0 Jul 10 02:28:42.738044 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:42.738008 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" event={"ID":"505dfb70-10af-47a4-9a81-46eb36e31291","Type":"ContainerDied","Data":"ce854d1dc98df1efcc97a7b6709d094fdb240759180c4fe753172fea2d1848ab"} Jul 10 02:28:42.739666 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:42.739647 2575 generic.go:356] "Generic (PLEG): container finished" podID="f5ff8126-0aee-4978-980a-f2ead93aef87" containerID="685b51a14b977783a249fdb567ea01d68d86dc61a6972934a1bb3652eb721c04" exitCode=0 Jul 10 02:28:42.739741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:42.739680 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" event={"ID":"f5ff8126-0aee-4978-980a-f2ead93aef87","Type":"ContainerDied","Data":"685b51a14b977783a249fdb567ea01d68d86dc61a6972934a1bb3652eb721c04"} Jul 10 02:28:42.739741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:42.739731 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" event={"ID":"f5ff8126-0aee-4978-980a-f2ead93aef87","Type":"ContainerStarted","Data":"8ab72374f8653a3a4059e1442a58138ad5b49f42da19d016ef69dfad3018dc46"} Jul 10 02:28:43.745803 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:43.745775 2575 generic.go:356] "Generic (PLEG): container finished" podID="e059e15a-f735-44cd-8584-b6e4fb85a7ee" containerID="a452e01716bf7cc63877c3565a17476d567fdf4b27a7f05462e4068ec05c321c" exitCode=0 Jul 10 02:28:43.746141 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:43.745843 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" event={"ID":"e059e15a-f735-44cd-8584-b6e4fb85a7ee","Type":"ContainerDied","Data":"a452e01716bf7cc63877c3565a17476d567fdf4b27a7f05462e4068ec05c321c"} Jul 10 02:28:44.752327 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:44.752293 2575 generic.go:356] "Generic (PLEG): container finished" podID="505dfb70-10af-47a4-9a81-46eb36e31291" containerID="37beed2412f14915d5c40d831466801901652c28cfcc2cd3a34a763c408cdc0d" exitCode=0 Jul 10 02:28:44.752735 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:44.752360 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" event={"ID":"505dfb70-10af-47a4-9a81-46eb36e31291","Type":"ContainerDied","Data":"37beed2412f14915d5c40d831466801901652c28cfcc2cd3a34a763c408cdc0d"} Jul 10 02:28:44.754591 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:44.754568 2575 generic.go:356] "Generic (PLEG): container finished" podID="e059e15a-f735-44cd-8584-b6e4fb85a7ee" containerID="25e40665b960c86089d4dbec11d81041e03b10cd3185bcf45bf26eacec387964" exitCode=0 Jul 10 02:28:44.754682 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:44.754643 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" event={"ID":"e059e15a-f735-44cd-8584-b6e4fb85a7ee","Type":"ContainerDied","Data":"25e40665b960c86089d4dbec11d81041e03b10cd3185bcf45bf26eacec387964"} Jul 10 02:28:44.756409 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:44.756388 2575 generic.go:356] "Generic (PLEG): container finished" podID="f5ff8126-0aee-4978-980a-f2ead93aef87" containerID="50e87327008f77bfca4c893978b35b3a490f521efd317eeffb9f685320d503e5" exitCode=0 Jul 10 02:28:44.756490 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:44.756469 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" event={"ID":"f5ff8126-0aee-4978-980a-f2ead93aef87","Type":"ContainerDied","Data":"50e87327008f77bfca4c893978b35b3a490f521efd317eeffb9f685320d503e5"} Jul 10 02:28:44.758463 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:44.758442 2575 generic.go:356] "Generic (PLEG): container finished" podID="61a2a0ec-b85e-4680-9d27-1dc6cc718f80" containerID="198ee2743134b3325425a4deaf1551ee97beb957437d5d17b63adc2fb87782f2" exitCode=0 Jul 10 02:28:44.758547 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:44.758481 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" event={"ID":"61a2a0ec-b85e-4680-9d27-1dc6cc718f80","Type":"ContainerDied","Data":"198ee2743134b3325425a4deaf1551ee97beb957437d5d17b63adc2fb87782f2"} Jul 10 02:28:45.764868 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:45.764831 2575 generic.go:356] "Generic (PLEG): container finished" podID="61a2a0ec-b85e-4680-9d27-1dc6cc718f80" containerID="033d11ced796e5caadad851a8936fbeedffd04c79400ca23541fbbcb86c8acdc" exitCode=0 Jul 10 02:28:45.765285 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:45.764917 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" event={"ID":"61a2a0ec-b85e-4680-9d27-1dc6cc718f80","Type":"ContainerDied","Data":"033d11ced796e5caadad851a8936fbeedffd04c79400ca23541fbbcb86c8acdc"} Jul 10 02:28:45.766882 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:45.766858 2575 generic.go:356] "Generic (PLEG): container finished" podID="505dfb70-10af-47a4-9a81-46eb36e31291" containerID="54b0382088e2ae134a919e24e0a6672d5a6745b590966ff665c2833a45f12b5d" exitCode=0 Jul 10 02:28:45.766966 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:45.766935 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" event={"ID":"505dfb70-10af-47a4-9a81-46eb36e31291","Type":"ContainerDied","Data":"54b0382088e2ae134a919e24e0a6672d5a6745b590966ff665c2833a45f12b5d"} Jul 10 02:28:45.768950 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:45.768928 2575 generic.go:356] "Generic (PLEG): container finished" podID="f5ff8126-0aee-4978-980a-f2ead93aef87" containerID="0613c4c674b120257ffb08d84ae6260678575415ef514c9b00f19b15a97c3d3a" exitCode=0 Jul 10 02:28:45.769056 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:45.769005 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" event={"ID":"f5ff8126-0aee-4978-980a-f2ead93aef87","Type":"ContainerDied","Data":"0613c4c674b120257ffb08d84ae6260678575415ef514c9b00f19b15a97c3d3a"} Jul 10 02:28:45.890851 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:45.890829 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" Jul 10 02:28:46.006030 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.005956 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92pkg\" (UniqueName: \"kubernetes.io/projected/e059e15a-f735-44cd-8584-b6e4fb85a7ee-kube-api-access-92pkg\") pod \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\" (UID: \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\") " Jul 10 02:28:46.006030 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.006014 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e059e15a-f735-44cd-8584-b6e4fb85a7ee-bundle\") pod \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\" (UID: \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\") " Jul 10 02:28:46.006200 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.006070 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e059e15a-f735-44cd-8584-b6e4fb85a7ee-util\") pod \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\" (UID: \"e059e15a-f735-44cd-8584-b6e4fb85a7ee\") " Jul 10 02:28:46.006543 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.006511 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e059e15a-f735-44cd-8584-b6e4fb85a7ee-bundle" (OuterVolumeSpecName: "bundle") pod "e059e15a-f735-44cd-8584-b6e4fb85a7ee" (UID: "e059e15a-f735-44cd-8584-b6e4fb85a7ee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:28:46.008042 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.008025 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e059e15a-f735-44cd-8584-b6e4fb85a7ee-kube-api-access-92pkg" (OuterVolumeSpecName: "kube-api-access-92pkg") pod "e059e15a-f735-44cd-8584-b6e4fb85a7ee" (UID: "e059e15a-f735-44cd-8584-b6e4fb85a7ee"). InnerVolumeSpecName "kube-api-access-92pkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:28:46.014823 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.014782 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e059e15a-f735-44cd-8584-b6e4fb85a7ee-util" (OuterVolumeSpecName: "util") pod "e059e15a-f735-44cd-8584-b6e4fb85a7ee" (UID: "e059e15a-f735-44cd-8584-b6e4fb85a7ee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:28:46.107043 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.107012 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e059e15a-f735-44cd-8584-b6e4fb85a7ee-util\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:28:46.107139 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.107048 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92pkg\" (UniqueName: \"kubernetes.io/projected/e059e15a-f735-44cd-8584-b6e4fb85a7ee-kube-api-access-92pkg\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:28:46.107139 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.107063 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e059e15a-f735-44cd-8584-b6e4fb85a7ee-bundle\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:28:46.774995 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.774962 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" Jul 10 02:28:46.775473 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.774996 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efjjd4k" event={"ID":"e059e15a-f735-44cd-8584-b6e4fb85a7ee","Type":"ContainerDied","Data":"4c0307d10d2c259bf51f86e49dd4912ec11ac31e7317451d7a210339e8574474"} Jul 10 02:28:46.775473 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.775047 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c0307d10d2c259bf51f86e49dd4912ec11ac31e7317451d7a210339e8574474" Jul 10 02:28:46.923149 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.923126 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" Jul 10 02:28:46.959776 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.959753 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" Jul 10 02:28:46.962778 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:46.962757 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" Jul 10 02:28:47.015086 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.015062 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5ff8126-0aee-4978-980a-f2ead93aef87-bundle\") pod \"f5ff8126-0aee-4978-980a-f2ead93aef87\" (UID: \"f5ff8126-0aee-4978-980a-f2ead93aef87\") " Jul 10 02:28:47.015228 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.015096 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5ff8126-0aee-4978-980a-f2ead93aef87-util\") pod \"f5ff8126-0aee-4978-980a-f2ead93aef87\" (UID: \"f5ff8126-0aee-4978-980a-f2ead93aef87\") " Jul 10 02:28:47.015228 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.015117 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/505dfb70-10af-47a4-9a81-46eb36e31291-bundle\") pod \"505dfb70-10af-47a4-9a81-46eb36e31291\" (UID: \"505dfb70-10af-47a4-9a81-46eb36e31291\") " Jul 10 02:28:47.015228 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.015136 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-bundle\") pod \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\" (UID: \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\") " Jul 10 02:28:47.015228 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.015161 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nbqb\" (UniqueName: \"kubernetes.io/projected/505dfb70-10af-47a4-9a81-46eb36e31291-kube-api-access-7nbqb\") pod \"505dfb70-10af-47a4-9a81-46eb36e31291\" (UID: \"505dfb70-10af-47a4-9a81-46eb36e31291\") " Jul 10 02:28:47.015228 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.015188 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv7pg\" (UniqueName: \"kubernetes.io/projected/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-kube-api-access-kv7pg\") pod \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\" (UID: \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\") " Jul 10 02:28:47.015228 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.015225 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-util\") pod \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\" (UID: \"61a2a0ec-b85e-4680-9d27-1dc6cc718f80\") " Jul 10 02:28:47.015562 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.015274 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/505dfb70-10af-47a4-9a81-46eb36e31291-util\") pod \"505dfb70-10af-47a4-9a81-46eb36e31291\" (UID: \"505dfb70-10af-47a4-9a81-46eb36e31291\") " Jul 10 02:28:47.015562 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.015302 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxwrh\" (UniqueName: \"kubernetes.io/projected/f5ff8126-0aee-4978-980a-f2ead93aef87-kube-api-access-lxwrh\") pod \"f5ff8126-0aee-4978-980a-f2ead93aef87\" (UID: \"f5ff8126-0aee-4978-980a-f2ead93aef87\") " Jul 10 02:28:47.015878 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.015843 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-bundle" (OuterVolumeSpecName: "bundle") pod "61a2a0ec-b85e-4680-9d27-1dc6cc718f80" (UID: "61a2a0ec-b85e-4680-9d27-1dc6cc718f80"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:28:47.015979 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.015950 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ff8126-0aee-4978-980a-f2ead93aef87-bundle" (OuterVolumeSpecName: "bundle") pod "f5ff8126-0aee-4978-980a-f2ead93aef87" (UID: "f5ff8126-0aee-4978-980a-f2ead93aef87"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:28:47.016053 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.016028 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505dfb70-10af-47a4-9a81-46eb36e31291-bundle" (OuterVolumeSpecName: "bundle") pod "505dfb70-10af-47a4-9a81-46eb36e31291" (UID: "505dfb70-10af-47a4-9a81-46eb36e31291"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:28:47.018149 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.018116 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-kube-api-access-kv7pg" (OuterVolumeSpecName: "kube-api-access-kv7pg") pod "61a2a0ec-b85e-4680-9d27-1dc6cc718f80" (UID: "61a2a0ec-b85e-4680-9d27-1dc6cc718f80"). InnerVolumeSpecName "kube-api-access-kv7pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:28:47.018428 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.018404 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ff8126-0aee-4978-980a-f2ead93aef87-kube-api-access-lxwrh" (OuterVolumeSpecName: "kube-api-access-lxwrh") pod "f5ff8126-0aee-4978-980a-f2ead93aef87" (UID: "f5ff8126-0aee-4978-980a-f2ead93aef87"). InnerVolumeSpecName "kube-api-access-lxwrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:28:47.018566 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.018548 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505dfb70-10af-47a4-9a81-46eb36e31291-kube-api-access-7nbqb" (OuterVolumeSpecName: "kube-api-access-7nbqb") pod "505dfb70-10af-47a4-9a81-46eb36e31291" (UID: "505dfb70-10af-47a4-9a81-46eb36e31291"). InnerVolumeSpecName "kube-api-access-7nbqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:28:47.022439 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.022401 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ff8126-0aee-4978-980a-f2ead93aef87-util" (OuterVolumeSpecName: "util") pod "f5ff8126-0aee-4978-980a-f2ead93aef87" (UID: "f5ff8126-0aee-4978-980a-f2ead93aef87"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:28:47.022533 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.022510 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-util" (OuterVolumeSpecName: "util") pod "61a2a0ec-b85e-4680-9d27-1dc6cc718f80" (UID: "61a2a0ec-b85e-4680-9d27-1dc6cc718f80"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:28:47.023008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.022978 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505dfb70-10af-47a4-9a81-46eb36e31291-util" (OuterVolumeSpecName: "util") pod "505dfb70-10af-47a4-9a81-46eb36e31291" (UID: "505dfb70-10af-47a4-9a81-46eb36e31291"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:28:47.116947 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.116918 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5ff8126-0aee-4978-980a-f2ead93aef87-bundle\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:28:47.116947 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.116948 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5ff8126-0aee-4978-980a-f2ead93aef87-util\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:28:47.117120 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.116957 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/505dfb70-10af-47a4-9a81-46eb36e31291-bundle\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:28:47.117120 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.116965 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-bundle\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:28:47.117120 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.116974 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nbqb\" (UniqueName: \"kubernetes.io/projected/505dfb70-10af-47a4-9a81-46eb36e31291-kube-api-access-7nbqb\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:28:47.117120 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.116985 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv7pg\" (UniqueName: \"kubernetes.io/projected/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-kube-api-access-kv7pg\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:28:47.117120 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.116993 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61a2a0ec-b85e-4680-9d27-1dc6cc718f80-util\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:28:47.117120 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.117003 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/505dfb70-10af-47a4-9a81-46eb36e31291-util\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:28:47.117120 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.117016 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxwrh\" (UniqueName: \"kubernetes.io/projected/f5ff8126-0aee-4978-980a-f2ead93aef87-kube-api-access-lxwrh\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:28:47.781089 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.781054 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" Jul 10 02:28:47.781089 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.781077 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff369pb46" event={"ID":"f5ff8126-0aee-4978-980a-f2ead93aef87","Type":"ContainerDied","Data":"8ab72374f8653a3a4059e1442a58138ad5b49f42da19d016ef69dfad3018dc46"} Jul 10 02:28:47.781630 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.781122 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ab72374f8653a3a4059e1442a58138ad5b49f42da19d016ef69dfad3018dc46" Jul 10 02:28:47.784087 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.784066 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" Jul 10 02:28:47.784087 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.784078 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabd8jl6" event={"ID":"61a2a0ec-b85e-4680-9d27-1dc6cc718f80","Type":"ContainerDied","Data":"7683ffa2afce5a9489e883aacc815cb6b039475f75408fe5829748f605a6cc4d"} Jul 10 02:28:47.784363 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.784119 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7683ffa2afce5a9489e883aacc815cb6b039475f75408fe5829748f605a6cc4d" Jul 10 02:28:47.786177 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.786152 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" event={"ID":"505dfb70-10af-47a4-9a81-46eb36e31291","Type":"ContainerDied","Data":"01d12b73ee82aab7b12f4cc76aa2b6436572306a69f2805068bac1b44838ec89"} Jul 10 02:28:47.786277 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.786197 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01d12b73ee82aab7b12f4cc76aa2b6436572306a69f2805068bac1b44838ec89" Jul 10 02:28:47.786277 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:47.786160 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76v7qvw" Jul 10 02:28:56.509455 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509419 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-558c6fb9c4-zhdr7"] Jul 10 02:28:56.509813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509720 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a2a0ec-b85e-4680-9d27-1dc6cc718f80" containerName="util" Jul 10 02:28:56.509813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509732 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a2a0ec-b85e-4680-9d27-1dc6cc718f80" containerName="util" Jul 10 02:28:56.509813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509739 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e059e15a-f735-44cd-8584-b6e4fb85a7ee" containerName="extract" Jul 10 02:28:56.509813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509745 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e059e15a-f735-44cd-8584-b6e4fb85a7ee" containerName="extract" Jul 10 02:28:56.509813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509756 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e059e15a-f735-44cd-8584-b6e4fb85a7ee" containerName="util" Jul 10 02:28:56.509813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509762 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e059e15a-f735-44cd-8584-b6e4fb85a7ee" containerName="util" Jul 10 02:28:56.509813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509770 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a2a0ec-b85e-4680-9d27-1dc6cc718f80" containerName="extract" Jul 10 02:28:56.509813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509776 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a2a0ec-b85e-4680-9d27-1dc6cc718f80" containerName="extract" Jul 10 02:28:56.509813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509787 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ff8126-0aee-4978-980a-f2ead93aef87" containerName="util" Jul 10 02:28:56.509813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509792 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ff8126-0aee-4978-980a-f2ead93aef87" containerName="util" Jul 10 02:28:56.509813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509799 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="505dfb70-10af-47a4-9a81-46eb36e31291" containerName="pull" Jul 10 02:28:56.509813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509804 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="505dfb70-10af-47a4-9a81-46eb36e31291" containerName="pull" Jul 10 02:28:56.509813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509813 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e059e15a-f735-44cd-8584-b6e4fb85a7ee" containerName="pull" Jul 10 02:28:56.509813 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509818 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e059e15a-f735-44cd-8584-b6e4fb85a7ee" containerName="pull" Jul 10 02:28:56.510266 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509824 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a2a0ec-b85e-4680-9d27-1dc6cc718f80" containerName="pull" Jul 10 02:28:56.510266 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509830 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a2a0ec-b85e-4680-9d27-1dc6cc718f80" containerName="pull" Jul 10 02:28:56.510266 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509837 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ff8126-0aee-4978-980a-f2ead93aef87" containerName="extract" Jul 10 02:28:56.510266 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509842 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ff8126-0aee-4978-980a-f2ead93aef87" containerName="extract" Jul 10 02:28:56.510266 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509848 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="505dfb70-10af-47a4-9a81-46eb36e31291" containerName="extract" Jul 10 02:28:56.510266 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509853 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="505dfb70-10af-47a4-9a81-46eb36e31291" containerName="extract" Jul 10 02:28:56.510266 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509866 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="505dfb70-10af-47a4-9a81-46eb36e31291" containerName="util" Jul 10 02:28:56.510266 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509871 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="505dfb70-10af-47a4-9a81-46eb36e31291" containerName="util" Jul 10 02:28:56.510266 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509877 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ff8126-0aee-4978-980a-f2ead93aef87" containerName="pull" Jul 10 02:28:56.510266 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509881 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ff8126-0aee-4978-980a-f2ead93aef87" containerName="pull" Jul 10 02:28:56.510266 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509928 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ff8126-0aee-4978-980a-f2ead93aef87" containerName="extract" Jul 10 02:28:56.510266 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509936 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e059e15a-f735-44cd-8584-b6e4fb85a7ee" containerName="extract" Jul 10 02:28:56.510266 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509941 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="505dfb70-10af-47a4-9a81-46eb36e31291" containerName="extract" Jul 10 02:28:56.510266 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.509948 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="61a2a0ec-b85e-4680-9d27-1dc6cc718f80" containerName="extract" Jul 10 02:28:56.512605 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.512588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-558c6fb9c4-zhdr7" Jul 10 02:28:56.516503 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.516481 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 02:28:56.516664 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.516646 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-krf9v\"" Jul 10 02:28:56.517051 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.517036 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 02:28:56.526498 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.526463 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-558c6fb9c4-zhdr7"] Jul 10 02:28:56.593038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.593004 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tts68\" (UniqueName: \"kubernetes.io/projected/6519b348-ea0d-4b30-8773-650b70714231-kube-api-access-tts68\") pod \"authorino-operator-558c6fb9c4-zhdr7\" (UID: \"6519b348-ea0d-4b30-8773-650b70714231\") " pod="kuadrant-system/authorino-operator-558c6fb9c4-zhdr7" Jul 10 02:28:56.694068 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.694030 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tts68\" (UniqueName: \"kubernetes.io/projected/6519b348-ea0d-4b30-8773-650b70714231-kube-api-access-tts68\") pod \"authorino-operator-558c6fb9c4-zhdr7\" (UID: \"6519b348-ea0d-4b30-8773-650b70714231\") " pod="kuadrant-system/authorino-operator-558c6fb9c4-zhdr7" Jul 10 02:28:56.706002 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.705966 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tts68\" (UniqueName: \"kubernetes.io/projected/6519b348-ea0d-4b30-8773-650b70714231-kube-api-access-tts68\") pod \"authorino-operator-558c6fb9c4-zhdr7\" (UID: \"6519b348-ea0d-4b30-8773-650b70714231\") " pod="kuadrant-system/authorino-operator-558c6fb9c4-zhdr7" Jul 10 02:28:56.821777 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.821681 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-558c6fb9c4-zhdr7" Jul 10 02:28:56.962143 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:56.962121 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-558c6fb9c4-zhdr7"] Jul 10 02:28:56.964476 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:28:56.964443 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6519b348_ea0d_4b30_8773_650b70714231.slice/crio-fc249bff3aafb9c572a73d85a4b4c78dc80f8c891621eef5b0596bb96510ee0b WatchSource:0}: Error finding container fc249bff3aafb9c572a73d85a4b4c78dc80f8c891621eef5b0596bb96510ee0b: Status 404 returned error can't find the container with id fc249bff3aafb9c572a73d85a4b4c78dc80f8c891621eef5b0596bb96510ee0b Jul 10 02:28:57.833947 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:57.833913 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-558c6fb9c4-zhdr7" event={"ID":"6519b348-ea0d-4b30-8773-650b70714231","Type":"ContainerStarted","Data":"fc249bff3aafb9c572a73d85a4b4c78dc80f8c891621eef5b0596bb96510ee0b"} Jul 10 02:28:58.726012 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:58.725975 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5546559d7-zvxmj"] Jul 10 02:28:58.728726 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:58.728687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5546559d7-zvxmj" Jul 10 02:28:58.732077 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:58.732024 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cb5m6\"" Jul 10 02:28:58.732077 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:58.732050 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 02:28:58.752929 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:58.752899 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5546559d7-zvxmj"] Jul 10 02:28:58.814878 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:58.814838 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77kzd\" (UniqueName: \"kubernetes.io/projected/a7e0c6db-86ef-4053-81dd-0b00748a676d-kube-api-access-77kzd\") pod \"dns-operator-controller-manager-5546559d7-zvxmj\" (UID: \"a7e0c6db-86ef-4053-81dd-0b00748a676d\") " pod="kuadrant-system/dns-operator-controller-manager-5546559d7-zvxmj" Jul 10 02:28:58.915601 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:58.915565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77kzd\" (UniqueName: \"kubernetes.io/projected/a7e0c6db-86ef-4053-81dd-0b00748a676d-kube-api-access-77kzd\") pod \"dns-operator-controller-manager-5546559d7-zvxmj\" (UID: \"a7e0c6db-86ef-4053-81dd-0b00748a676d\") " pod="kuadrant-system/dns-operator-controller-manager-5546559d7-zvxmj" Jul 10 02:28:58.930193 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:58.930162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77kzd\" (UniqueName: \"kubernetes.io/projected/a7e0c6db-86ef-4053-81dd-0b00748a676d-kube-api-access-77kzd\") pod \"dns-operator-controller-manager-5546559d7-zvxmj\" (UID: \"a7e0c6db-86ef-4053-81dd-0b00748a676d\") " pod="kuadrant-system/dns-operator-controller-manager-5546559d7-zvxmj" Jul 10 02:28:59.041330 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:59.041238 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5546559d7-zvxmj" Jul 10 02:28:59.920823 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:28:59.920797 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5546559d7-zvxmj"] Jul 10 02:28:59.921276 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:28:59.921235 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e0c6db_86ef_4053_81dd_0b00748a676d.slice/crio-ab68c954d28c68440084cbba424e5e8d07f315a3e491a94538e15499289b2eeb WatchSource:0}: Error finding container ab68c954d28c68440084cbba424e5e8d07f315a3e491a94538e15499289b2eeb: Status 404 returned error can't find the container with id ab68c954d28c68440084cbba424e5e8d07f315a3e491a94538e15499289b2eeb Jul 10 02:29:00.858382 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:00.858335 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5546559d7-zvxmj" event={"ID":"a7e0c6db-86ef-4053-81dd-0b00748a676d","Type":"ContainerStarted","Data":"ab68c954d28c68440084cbba424e5e8d07f315a3e491a94538e15499289b2eeb"} Jul 10 02:29:00.859891 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:00.859862 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-558c6fb9c4-zhdr7" event={"ID":"6519b348-ea0d-4b30-8773-650b70714231","Type":"ContainerStarted","Data":"97ba68e8c4c97f123b07fdba1d7ace18620499ab9949f5ccbf493c48fa5c7379"} Jul 10 02:29:00.860025 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:00.860006 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-558c6fb9c4-zhdr7" Jul 10 02:29:00.887128 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:00.887074 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-558c6fb9c4-zhdr7" podStartSLOduration=2.021682104 podStartE2EDuration="4.887061905s" totalImagesPullingTime="2.865379801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:29:00.885638513 +0000 UTC m=+370.285091426" watchObservedRunningTime="2026-07-10 02:29:00.887061905 +0000 UTC m=+370.286514817" Jul 10 02:29:03.235127 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:03.235091 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr"] Jul 10 02:29:03.237452 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:03.237435 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:03.240754 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:03.240730 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x245d\"" Jul 10 02:29:03.258507 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:03.258482 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr"] Jul 10 02:29:03.354964 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:03.354920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4mfp\" (UniqueName: \"kubernetes.io/projected/c7f84e05-e317-49dc-baf2-d7a9d92bd342-kube-api-access-s4mfp\") pod \"kuadrant-operator-controller-manager-6c647fc79-jwpqr\" (UID: \"c7f84e05-e317-49dc-baf2-d7a9d92bd342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:03.355388 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:03.355353 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7f84e05-e317-49dc-baf2-d7a9d92bd342-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6c647fc79-jwpqr\" (UID: \"c7f84e05-e317-49dc-baf2-d7a9d92bd342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:03.456456 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:03.456418 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4mfp\" (UniqueName: \"kubernetes.io/projected/c7f84e05-e317-49dc-baf2-d7a9d92bd342-kube-api-access-s4mfp\") pod \"kuadrant-operator-controller-manager-6c647fc79-jwpqr\" (UID: \"c7f84e05-e317-49dc-baf2-d7a9d92bd342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:03.456614 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:03.456465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7f84e05-e317-49dc-baf2-d7a9d92bd342-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6c647fc79-jwpqr\" (UID: \"c7f84e05-e317-49dc-baf2-d7a9d92bd342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:03.456944 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:03.456924 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7f84e05-e317-49dc-baf2-d7a9d92bd342-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6c647fc79-jwpqr\" (UID: \"c7f84e05-e317-49dc-baf2-d7a9d92bd342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:03.469890 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:03.469859 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4mfp\" (UniqueName: \"kubernetes.io/projected/c7f84e05-e317-49dc-baf2-d7a9d92bd342-kube-api-access-s4mfp\") pod \"kuadrant-operator-controller-manager-6c647fc79-jwpqr\" (UID: \"c7f84e05-e317-49dc-baf2-d7a9d92bd342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:03.548083 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:03.548003 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:03.690524 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:03.690497 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr"] Jul 10 02:29:03.692412 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:29:03.692364 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f84e05_e317_49dc_baf2_d7a9d92bd342.slice/crio-a24fe181d8091a77ef7a64a76ec33cc22847a835c7b7bf58049eca54fdb83790 WatchSource:0}: Error finding container a24fe181d8091a77ef7a64a76ec33cc22847a835c7b7bf58049eca54fdb83790: Status 404 returned error can't find the container with id a24fe181d8091a77ef7a64a76ec33cc22847a835c7b7bf58049eca54fdb83790 Jul 10 02:29:03.875653 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:03.875618 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" event={"ID":"c7f84e05-e317-49dc-baf2-d7a9d92bd342","Type":"ContainerStarted","Data":"a24fe181d8091a77ef7a64a76ec33cc22847a835c7b7bf58049eca54fdb83790"} Jul 10 02:29:05.230916 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:05.230879 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk"] Jul 10 02:29:05.233512 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:05.233489 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" Jul 10 02:29:05.237493 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:05.237463 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nldv8\"" Jul 10 02:29:05.251874 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:05.251844 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk"] Jul 10 02:29:05.373948 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:05.373911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndfb9\" (UniqueName: \"kubernetes.io/projected/4d045df6-5a38-4135-bb7a-a9f393ca38d2-kube-api-access-ndfb9\") pod \"limitador-operator-controller-manager-7d6bb9cf99-hhffk\" (UID: \"4d045df6-5a38-4135-bb7a-a9f393ca38d2\") " pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" Jul 10 02:29:05.475232 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:05.475193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndfb9\" (UniqueName: \"kubernetes.io/projected/4d045df6-5a38-4135-bb7a-a9f393ca38d2-kube-api-access-ndfb9\") pod \"limitador-operator-controller-manager-7d6bb9cf99-hhffk\" (UID: \"4d045df6-5a38-4135-bb7a-a9f393ca38d2\") " pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" Jul 10 02:29:05.484512 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:05.484444 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndfb9\" (UniqueName: \"kubernetes.io/projected/4d045df6-5a38-4135-bb7a-a9f393ca38d2-kube-api-access-ndfb9\") pod \"limitador-operator-controller-manager-7d6bb9cf99-hhffk\" (UID: \"4d045df6-5a38-4135-bb7a-a9f393ca38d2\") " pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" Jul 10 02:29:05.546974 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:05.546936 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" Jul 10 02:29:05.699208 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:05.699155 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk"] Jul 10 02:29:05.701204 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:29:05.701169 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d045df6_5a38_4135_bb7a_a9f393ca38d2.slice/crio-7a78046101d18778b01a8a8600057ecacb0ab91da692d69372a65c6af1075864 WatchSource:0}: Error finding container 7a78046101d18778b01a8a8600057ecacb0ab91da692d69372a65c6af1075864: Status 404 returned error can't find the container with id 7a78046101d18778b01a8a8600057ecacb0ab91da692d69372a65c6af1075864 Jul 10 02:29:05.889222 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:05.889177 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" event={"ID":"4d045df6-5a38-4135-bb7a-a9f393ca38d2","Type":"ContainerStarted","Data":"7a78046101d18778b01a8a8600057ecacb0ab91da692d69372a65c6af1075864"} Jul 10 02:29:07.901014 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:07.900979 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" event={"ID":"4d045df6-5a38-4135-bb7a-a9f393ca38d2","Type":"ContainerStarted","Data":"3a4b55ce470338b1e36e4e925caaf0cf4128732ae02db5b043ede3d951b5d257"} Jul 10 02:29:07.901390 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:07.901057 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" Jul 10 02:29:07.925076 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:07.925025 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" podStartSLOduration=1.316338386 podStartE2EDuration="2.925009858s" totalImagesPullingTime="1.608671472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:29:07.923476063 +0000 UTC m=+377.322928975" watchObservedRunningTime="2026-07-10 02:29:07.925009858 +0000 UTC m=+377.324462771" Jul 10 02:29:11.867162 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:11.867131 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-558c6fb9c4-zhdr7" Jul 10 02:29:11.924586 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:11.924503 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" event={"ID":"c7f84e05-e317-49dc-baf2-d7a9d92bd342","Type":"ContainerStarted","Data":"5c6d0528ec0bf6d3e4958cf028c9a656201e247f9bb9a1547ba753c0a6b3b4da"} Jul 10 02:29:11.924586 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:11.924586 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:11.945478 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:11.945427 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" podStartSLOduration=1.017524893 podStartE2EDuration="8.945408377s" totalImagesPullingTime="7.927883484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:29:11.944758721 +0000 UTC m=+381.344211635" watchObservedRunningTime="2026-07-10 02:29:11.945408377 +0000 UTC m=+381.344861292" Jul 10 02:29:14.308337 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:14.308302 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv"] Jul 10 02:29:14.310605 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:14.310584 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" Jul 10 02:29:14.313410 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:14.313370 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 02:29:14.313829 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:14.313809 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qpsps\"" Jul 10 02:29:14.313879 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:14.313837 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 02:29:14.324383 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:14.324360 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv"] Jul 10 02:29:14.454071 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:14.454036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a3a9346d-6628-43f8-85af-e6e451b7d156-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-24gbv\" (UID: \"a3a9346d-6628-43f8-85af-e6e451b7d156\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" Jul 10 02:29:14.454235 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:14.454091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xphwf\" (UniqueName: \"kubernetes.io/projected/a3a9346d-6628-43f8-85af-e6e451b7d156-kube-api-access-xphwf\") pod \"kuadrant-console-plugin-85fd4b8b9d-24gbv\" (UID: \"a3a9346d-6628-43f8-85af-e6e451b7d156\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" Jul 10 02:29:14.454235 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:14.454191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a9346d-6628-43f8-85af-e6e451b7d156-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-24gbv\" (UID: \"a3a9346d-6628-43f8-85af-e6e451b7d156\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" Jul 10 02:29:14.555546 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:14.555509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xphwf\" (UniqueName: \"kubernetes.io/projected/a3a9346d-6628-43f8-85af-e6e451b7d156-kube-api-access-xphwf\") pod \"kuadrant-console-plugin-85fd4b8b9d-24gbv\" (UID: \"a3a9346d-6628-43f8-85af-e6e451b7d156\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" Jul 10 02:29:14.555714 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:14.555580 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a9346d-6628-43f8-85af-e6e451b7d156-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-24gbv\" (UID: \"a3a9346d-6628-43f8-85af-e6e451b7d156\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" Jul 10 02:29:14.555714 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:14.555612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a3a9346d-6628-43f8-85af-e6e451b7d156-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-24gbv\" (UID: \"a3a9346d-6628-43f8-85af-e6e451b7d156\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" Jul 10 02:29:14.555786 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:29:14.555737 2575 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 10 02:29:14.555838 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:29:14.555827 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3a9346d-6628-43f8-85af-e6e451b7d156-plugin-serving-cert podName:a3a9346d-6628-43f8-85af-e6e451b7d156 nodeName:}" failed. No retries permitted until 2026-07-10 02:29:15.055804739 +0000 UTC m=+384.455257634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a3a9346d-6628-43f8-85af-e6e451b7d156-plugin-serving-cert") pod "kuadrant-console-plugin-85fd4b8b9d-24gbv" (UID: "a3a9346d-6628-43f8-85af-e6e451b7d156") : secret "plugin-serving-cert" not found Jul 10 02:29:14.556193 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:14.556176 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a3a9346d-6628-43f8-85af-e6e451b7d156-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-24gbv\" (UID: \"a3a9346d-6628-43f8-85af-e6e451b7d156\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" Jul 10 02:29:14.565350 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:14.565283 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xphwf\" (UniqueName: \"kubernetes.io/projected/a3a9346d-6628-43f8-85af-e6e451b7d156-kube-api-access-xphwf\") pod \"kuadrant-console-plugin-85fd4b8b9d-24gbv\" (UID: \"a3a9346d-6628-43f8-85af-e6e451b7d156\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" Jul 10 02:29:15.060597 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:15.060558 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a9346d-6628-43f8-85af-e6e451b7d156-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-24gbv\" (UID: \"a3a9346d-6628-43f8-85af-e6e451b7d156\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" Jul 10 02:29:15.062967 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:15.062936 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a9346d-6628-43f8-85af-e6e451b7d156-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-24gbv\" (UID: \"a3a9346d-6628-43f8-85af-e6e451b7d156\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" Jul 10 02:29:15.221374 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:15.221337 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" Jul 10 02:29:15.369127 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:15.369103 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv"] Jul 10 02:29:15.371193 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:29:15.371166 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3a9346d_6628_43f8_85af_e6e451b7d156.slice/crio-5a7b80b9b5ad260264ec9d526b3e809c42a4057df3d50877ebaf9d48ca2dcff0 WatchSource:0}: Error finding container 5a7b80b9b5ad260264ec9d526b3e809c42a4057df3d50877ebaf9d48ca2dcff0: Status 404 returned error can't find the container with id 5a7b80b9b5ad260264ec9d526b3e809c42a4057df3d50877ebaf9d48ca2dcff0 Jul 10 02:29:15.945412 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:15.945376 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" event={"ID":"a3a9346d-6628-43f8-85af-e6e451b7d156","Type":"ContainerStarted","Data":"5a7b80b9b5ad260264ec9d526b3e809c42a4057df3d50877ebaf9d48ca2dcff0"} Jul 10 02:29:16.952580 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:16.952490 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5546559d7-zvxmj" event={"ID":"a7e0c6db-86ef-4053-81dd-0b00748a676d","Type":"ContainerStarted","Data":"ccb5aad3fdad0be36971c4e94fdcb9c8871caea43d3baa513c77da26ab9a2e77"} Jul 10 02:29:16.952967 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:16.952617 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5546559d7-zvxmj" Jul 10 02:29:16.978312 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:16.978259 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5546559d7-zvxmj" podStartSLOduration=2.245786295 podStartE2EDuration="18.978228913s" totalImagesPullingTime="16.732442618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:29:16.977814996 +0000 UTC m=+386.377267912" watchObservedRunningTime="2026-07-10 02:29:16.978228913 +0000 UTC m=+386.377681825" Jul 10 02:29:18.908566 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:18.908539 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" Jul 10 02:29:22.932532 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:22.932500 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:24.730115 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.730031 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr"] Jul 10 02:29:24.730529 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.730235 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" podUID="c7f84e05-e317-49dc-baf2-d7a9d92bd342" containerName="manager" containerID="cri-o://5c6d0528ec0bf6d3e4958cf028c9a656201e247f9bb9a1547ba753c0a6b3b4da" gracePeriod=2 Jul 10 02:29:24.740281 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.740229 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr"] Jul 10 02:29:24.760826 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.760799 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966"] Jul 10 02:29:24.761124 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.761113 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f84e05-e317-49dc-baf2-d7a9d92bd342" containerName="manager" Jul 10 02:29:24.761161 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.761126 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f84e05-e317-49dc-baf2-d7a9d92bd342" containerName="manager" Jul 10 02:29:24.761206 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.761185 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f84e05-e317-49dc-baf2-d7a9d92bd342" containerName="manager" Jul 10 02:29:24.764196 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.764178 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" Jul 10 02:29:24.782604 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.782575 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk"] Jul 10 02:29:24.782838 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.782786 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" podUID="4d045df6-5a38-4135-bb7a-a9f393ca38d2" containerName="manager" containerID="cri-o://3a4b55ce470338b1e36e4e925caaf0cf4128732ae02db5b043ede3d951b5d257" gracePeriod=2 Jul 10 02:29:24.784901 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.784869 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966"] Jul 10 02:29:24.796324 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.796297 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk"] Jul 10 02:29:24.803920 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:29:24.803887 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6c647fc79-jwpqr\" is forbidden: User \"system:node:ip-10-0-128-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-69.ec2.internal' and this object" podUID="c7f84e05-e317-49dc-baf2-d7a9d92bd342" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:24.806059 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:29:24.806030 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6c647fc79-jwpqr\" is forbidden: User \"system:node:ip-10-0-128-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-69.ec2.internal' and this object" podUID="c7f84e05-e317-49dc-baf2-d7a9d92bd342" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:24.808577 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:29:24.808546 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-7d6bb9cf99-hhffk\" is forbidden: User \"system:node:ip-10-0-128-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-69.ec2.internal' and this object" podUID="4d045df6-5a38-4135-bb7a-a9f393ca38d2" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" Jul 10 02:29:24.810490 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.810468 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-7jlv4"] Jul 10 02:29:24.810916 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.810902 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d045df6-5a38-4135-bb7a-a9f393ca38d2" containerName="manager" Jul 10 02:29:24.810980 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.810919 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d045df6-5a38-4135-bb7a-a9f393ca38d2" containerName="manager" Jul 10 02:29:24.811034 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.811008 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d045df6-5a38-4135-bb7a-a9f393ca38d2" containerName="manager" Jul 10 02:29:24.815647 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.815630 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-7jlv4" Jul 10 02:29:24.839895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.839867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtl9\" (UniqueName: \"kubernetes.io/projected/bccd1c52-a5e6-4a92-b128-ed93714208a9-kube-api-access-9mtl9\") pod \"kuadrant-operator-controller-manager-6c647fc79-l2966\" (UID: \"bccd1c52-a5e6-4a92-b128-ed93714208a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" Jul 10 02:29:24.840026 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.839946 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bccd1c52-a5e6-4a92-b128-ed93714208a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6c647fc79-l2966\" (UID: \"bccd1c52-a5e6-4a92-b128-ed93714208a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" Jul 10 02:29:24.840026 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.839987 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m92w2\" (UniqueName: \"kubernetes.io/projected/4f6be0eb-9bac-4347-857a-b88919001b05-kube-api-access-m92w2\") pod \"limitador-operator-controller-manager-7d6bb9cf99-7jlv4\" (UID: \"4f6be0eb-9bac-4347-857a-b88919001b05\") " pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-7jlv4" Jul 10 02:29:24.847771 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.847743 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-7jlv4"] Jul 10 02:29:24.883337 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:29:24.883302 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6c647fc79-jwpqr\" is forbidden: User \"system:node:ip-10-0-128-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-69.ec2.internal' and this object" podUID="c7f84e05-e317-49dc-baf2-d7a9d92bd342" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:24.888464 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:29:24.888438 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-7d6bb9cf99-hhffk\" is forbidden: User \"system:node:ip-10-0-128-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-69.ec2.internal' and this object" podUID="4d045df6-5a38-4135-bb7a-a9f393ca38d2" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" Jul 10 02:29:24.940884 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.940849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bccd1c52-a5e6-4a92-b128-ed93714208a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6c647fc79-l2966\" (UID: \"bccd1c52-a5e6-4a92-b128-ed93714208a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" Jul 10 02:29:24.941059 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.940895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m92w2\" (UniqueName: \"kubernetes.io/projected/4f6be0eb-9bac-4347-857a-b88919001b05-kube-api-access-m92w2\") pod \"limitador-operator-controller-manager-7d6bb9cf99-7jlv4\" (UID: \"4f6be0eb-9bac-4347-857a-b88919001b05\") " pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-7jlv4" Jul 10 02:29:24.941059 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.940960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtl9\" (UniqueName: \"kubernetes.io/projected/bccd1c52-a5e6-4a92-b128-ed93714208a9-kube-api-access-9mtl9\") pod \"kuadrant-operator-controller-manager-6c647fc79-l2966\" (UID: \"bccd1c52-a5e6-4a92-b128-ed93714208a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" Jul 10 02:29:24.941324 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.941295 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bccd1c52-a5e6-4a92-b128-ed93714208a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6c647fc79-l2966\" (UID: \"bccd1c52-a5e6-4a92-b128-ed93714208a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" Jul 10 02:29:24.963221 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.963072 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m92w2\" (UniqueName: \"kubernetes.io/projected/4f6be0eb-9bac-4347-857a-b88919001b05-kube-api-access-m92w2\") pod \"limitador-operator-controller-manager-7d6bb9cf99-7jlv4\" (UID: \"4f6be0eb-9bac-4347-857a-b88919001b05\") " pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-7jlv4" Jul 10 02:29:24.965186 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.965160 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtl9\" (UniqueName: \"kubernetes.io/projected/bccd1c52-a5e6-4a92-b128-ed93714208a9-kube-api-access-9mtl9\") pod \"kuadrant-operator-controller-manager-6c647fc79-l2966\" (UID: \"bccd1c52-a5e6-4a92-b128-ed93714208a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" Jul 10 02:29:24.999170 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:24.999135 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" event={"ID":"a3a9346d-6628-43f8-85af-e6e451b7d156","Type":"ContainerStarted","Data":"32a7e69f7b1b58c649935bd14234f03d31bacb243d0504d4684fa2b7a98f188a"} Jul 10 02:29:25.001138 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.001112 2575 generic.go:356] "Generic (PLEG): container finished" podID="4d045df6-5a38-4135-bb7a-a9f393ca38d2" containerID="3a4b55ce470338b1e36e4e925caaf0cf4128732ae02db5b043ede3d951b5d257" exitCode=0 Jul 10 02:29:25.002284 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:29:25.002239 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6c647fc79-jwpqr\" is forbidden: User \"system:node:ip-10-0-128-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-69.ec2.internal' and this object" podUID="c7f84e05-e317-49dc-baf2-d7a9d92bd342" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:25.002702 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.002678 2575 generic.go:356] "Generic (PLEG): container finished" podID="c7f84e05-e317-49dc-baf2-d7a9d92bd342" containerID="5c6d0528ec0bf6d3e4958cf028c9a656201e247f9bb9a1547ba753c0a6b3b4da" exitCode=0 Jul 10 02:29:25.007010 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.006982 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:25.014373 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.014353 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" Jul 10 02:29:25.026770 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:29:25.026743 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-7d6bb9cf99-hhffk\" is forbidden: User \"system:node:ip-10-0-128-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-69.ec2.internal' and this object" podUID="4d045df6-5a38-4135-bb7a-a9f393ca38d2" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" Jul 10 02:29:25.027619 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.027584 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-24gbv" podStartSLOduration=2.018954956 podStartE2EDuration="11.0275707s" totalImagesPullingTime="9.008615744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:29:25.02478542 +0000 UTC m=+394.424238335" watchObservedRunningTime="2026-07-10 02:29:25.0275707 +0000 UTC m=+394.427023613" Jul 10 02:29:25.028911 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:29:25.028883 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6c647fc79-jwpqr\" is forbidden: User \"system:node:ip-10-0-128-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-69.ec2.internal' and this object" podUID="c7f84e05-e317-49dc-baf2-d7a9d92bd342" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:25.033551 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:29:25.033529 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-7d6bb9cf99-hhffk\" is forbidden: User \"system:node:ip-10-0-128-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-69.ec2.internal' and this object" podUID="4d045df6-5a38-4135-bb7a-a9f393ca38d2" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" Jul 10 02:29:25.041832 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.041808 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndfb9\" (UniqueName: \"kubernetes.io/projected/4d045df6-5a38-4135-bb7a-a9f393ca38d2-kube-api-access-ndfb9\") pod \"4d045df6-5a38-4135-bb7a-a9f393ca38d2\" (UID: \"4d045df6-5a38-4135-bb7a-a9f393ca38d2\") " Jul 10 02:29:25.041970 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.041898 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7f84e05-e317-49dc-baf2-d7a9d92bd342-extensions-socket-volume\") pod \"c7f84e05-e317-49dc-baf2-d7a9d92bd342\" (UID: \"c7f84e05-e317-49dc-baf2-d7a9d92bd342\") " Jul 10 02:29:25.042038 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.041978 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4mfp\" (UniqueName: \"kubernetes.io/projected/c7f84e05-e317-49dc-baf2-d7a9d92bd342-kube-api-access-s4mfp\") pod \"c7f84e05-e317-49dc-baf2-d7a9d92bd342\" (UID: \"c7f84e05-e317-49dc-baf2-d7a9d92bd342\") " Jul 10 02:29:25.042509 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.042470 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f84e05-e317-49dc-baf2-d7a9d92bd342-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c7f84e05-e317-49dc-baf2-d7a9d92bd342" (UID: "c7f84e05-e317-49dc-baf2-d7a9d92bd342"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:29:25.044222 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.044188 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f84e05-e317-49dc-baf2-d7a9d92bd342-kube-api-access-s4mfp" (OuterVolumeSpecName: "kube-api-access-s4mfp") pod "c7f84e05-e317-49dc-baf2-d7a9d92bd342" (UID: "c7f84e05-e317-49dc-baf2-d7a9d92bd342"). InnerVolumeSpecName "kube-api-access-s4mfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:29:25.044473 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.044455 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d045df6-5a38-4135-bb7a-a9f393ca38d2-kube-api-access-ndfb9" (OuterVolumeSpecName: "kube-api-access-ndfb9") pod "4d045df6-5a38-4135-bb7a-a9f393ca38d2" (UID: "4d045df6-5a38-4135-bb7a-a9f393ca38d2"). InnerVolumeSpecName "kube-api-access-ndfb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:29:25.129640 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.129610 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d045df6-5a38-4135-bb7a-a9f393ca38d2" path="/var/lib/kubelet/pods/4d045df6-5a38-4135-bb7a-a9f393ca38d2/volumes" Jul 10 02:29:25.129980 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.129966 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7f84e05-e317-49dc-baf2-d7a9d92bd342" path="/var/lib/kubelet/pods/c7f84e05-e317-49dc-baf2-d7a9d92bd342/volumes" Jul 10 02:29:25.142832 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.142809 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7f84e05-e317-49dc-baf2-d7a9d92bd342-extensions-socket-volume\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:29:25.142832 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.142831 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4mfp\" (UniqueName: \"kubernetes.io/projected/c7f84e05-e317-49dc-baf2-d7a9d92bd342-kube-api-access-s4mfp\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:29:25.142956 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.142854 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndfb9\" (UniqueName: \"kubernetes.io/projected/4d045df6-5a38-4135-bb7a-a9f393ca38d2-kube-api-access-ndfb9\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:29:25.167776 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.167755 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" Jul 10 02:29:25.173852 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.173831 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-7jlv4" Jul 10 02:29:25.332005 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.331981 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966"] Jul 10 02:29:25.334589 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:29:25.334565 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbccd1c52_a5e6_4a92_b128_ed93714208a9.slice/crio-e36adfbf05d3e0bb0ff1e3a156adfa04c459799c72084f7f2b0fa517959a4f71 WatchSource:0}: Error finding container e36adfbf05d3e0bb0ff1e3a156adfa04c459799c72084f7f2b0fa517959a4f71: Status 404 returned error can't find the container with id e36adfbf05d3e0bb0ff1e3a156adfa04c459799c72084f7f2b0fa517959a4f71 Jul 10 02:29:25.353833 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:25.353808 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-7jlv4"] Jul 10 02:29:25.356834 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:29:25.356808 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6be0eb_9bac_4347_857a_b88919001b05.slice/crio-77907e711f27fc084cc8f23ead451c8dae1d8a9b5ea1aa7d92d4d0fc542f50e2 WatchSource:0}: Error finding container 77907e711f27fc084cc8f23ead451c8dae1d8a9b5ea1aa7d92d4d0fc542f50e2: Status 404 returned error can't find the container with id 77907e711f27fc084cc8f23ead451c8dae1d8a9b5ea1aa7d92d4d0fc542f50e2 Jul 10 02:29:26.008878 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:26.008849 2575 scope.go:117] "RemoveContainer" containerID="3a4b55ce470338b1e36e4e925caaf0cf4128732ae02db5b043ede3d951b5d257" Jul 10 02:29:26.009328 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:26.008853 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-hhffk" Jul 10 02:29:26.010710 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:26.010683 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-7jlv4" event={"ID":"4f6be0eb-9bac-4347-857a-b88919001b05","Type":"ContainerStarted","Data":"7f67dc399f0aebacea572893ca75131cdc926838c75dfe085ed6d2122b456418"} Jul 10 02:29:26.010843 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:26.010741 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-7jlv4" event={"ID":"4f6be0eb-9bac-4347-857a-b88919001b05","Type":"ContainerStarted","Data":"77907e711f27fc084cc8f23ead451c8dae1d8a9b5ea1aa7d92d4d0fc542f50e2"} Jul 10 02:29:26.011052 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:26.011019 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-7jlv4" Jul 10 02:29:26.012272 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:26.012236 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-jwpqr" Jul 10 02:29:26.014164 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:26.014138 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" event={"ID":"bccd1c52-a5e6-4a92-b128-ed93714208a9","Type":"ContainerStarted","Data":"93f92915002f3fce93e3d65ca4bc1408c262ac10a73759528fd27ea3a7f173c2"} Jul 10 02:29:26.014273 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:26.014192 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" event={"ID":"bccd1c52-a5e6-4a92-b128-ed93714208a9","Type":"ContainerStarted","Data":"e36adfbf05d3e0bb0ff1e3a156adfa04c459799c72084f7f2b0fa517959a4f71"} Jul 10 02:29:26.014363 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:26.014346 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" Jul 10 02:29:26.018860 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:26.018843 2575 scope.go:117] "RemoveContainer" containerID="5c6d0528ec0bf6d3e4958cf028c9a656201e247f9bb9a1547ba753c0a6b3b4da" Jul 10 02:29:26.038336 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:26.038296 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-7jlv4" podStartSLOduration=2.038281543 podStartE2EDuration="2.038281543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:29:26.037907645 +0000 UTC m=+395.437360570" watchObservedRunningTime="2026-07-10 02:29:26.038281543 +0000 UTC m=+395.437734453" Jul 10 02:29:26.063886 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:26.063833 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" podStartSLOduration=2.063818382 podStartE2EDuration="2.063818382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:29:26.063345297 +0000 UTC m=+395.462798209" watchObservedRunningTime="2026-07-10 02:29:26.063818382 +0000 UTC m=+395.463271295" Jul 10 02:29:27.959399 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:27.959366 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5546559d7-zvxmj" Jul 10 02:29:37.023735 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:37.023703 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-7jlv4" Jul 10 02:29:37.024152 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:37.023869 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" Jul 10 02:29:45.980824 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:45.980785 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966"] Jul 10 02:29:45.981231 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:45.980984 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" podUID="bccd1c52-a5e6-4a92-b128-ed93714208a9" containerName="manager" containerID="cri-o://93f92915002f3fce93e3d65ca4bc1408c262ac10a73759528fd27ea3a7f173c2" gracePeriod=10 Jul 10 02:29:46.124195 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.124156 2575 generic.go:356] "Generic (PLEG): container finished" podID="bccd1c52-a5e6-4a92-b128-ed93714208a9" containerID="93f92915002f3fce93e3d65ca4bc1408c262ac10a73759528fd27ea3a7f173c2" exitCode=0 Jul 10 02:29:46.124379 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.124238 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" event={"ID":"bccd1c52-a5e6-4a92-b128-ed93714208a9","Type":"ContainerDied","Data":"93f92915002f3fce93e3d65ca4bc1408c262ac10a73759528fd27ea3a7f173c2"} Jul 10 02:29:46.210480 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.210450 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8"] Jul 10 02:29:46.212784 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.212768 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" Jul 10 02:29:46.227706 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.227688 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" Jul 10 02:29:46.238063 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.238034 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8"] Jul 10 02:29:46.318663 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.318633 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mtl9\" (UniqueName: \"kubernetes.io/projected/bccd1c52-a5e6-4a92-b128-ed93714208a9-kube-api-access-9mtl9\") pod \"bccd1c52-a5e6-4a92-b128-ed93714208a9\" (UID: \"bccd1c52-a5e6-4a92-b128-ed93714208a9\") " Jul 10 02:29:46.318825 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.318671 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bccd1c52-a5e6-4a92-b128-ed93714208a9-extensions-socket-volume\") pod \"bccd1c52-a5e6-4a92-b128-ed93714208a9\" (UID: \"bccd1c52-a5e6-4a92-b128-ed93714208a9\") " Jul 10 02:29:46.318825 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.318768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77h99\" (UniqueName: \"kubernetes.io/projected/51e434b3-e3ff-4224-864f-deb2575ec12c-kube-api-access-77h99\") pod \"kuadrant-operator-controller-manager-7569bc7c69-rsrh8\" (UID: \"51e434b3-e3ff-4224-864f-deb2575ec12c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" Jul 10 02:29:46.318920 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.318824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51e434b3-e3ff-4224-864f-deb2575ec12c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7569bc7c69-rsrh8\" (UID: \"51e434b3-e3ff-4224-864f-deb2575ec12c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" Jul 10 02:29:46.319086 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.319054 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bccd1c52-a5e6-4a92-b128-ed93714208a9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bccd1c52-a5e6-4a92-b128-ed93714208a9" (UID: "bccd1c52-a5e6-4a92-b128-ed93714208a9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:29:46.320681 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.320659 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccd1c52-a5e6-4a92-b128-ed93714208a9-kube-api-access-9mtl9" (OuterVolumeSpecName: "kube-api-access-9mtl9") pod "bccd1c52-a5e6-4a92-b128-ed93714208a9" (UID: "bccd1c52-a5e6-4a92-b128-ed93714208a9"). InnerVolumeSpecName "kube-api-access-9mtl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:29:46.419384 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.419341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77h99\" (UniqueName: \"kubernetes.io/projected/51e434b3-e3ff-4224-864f-deb2575ec12c-kube-api-access-77h99\") pod \"kuadrant-operator-controller-manager-7569bc7c69-rsrh8\" (UID: \"51e434b3-e3ff-4224-864f-deb2575ec12c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" Jul 10 02:29:46.419583 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.419407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51e434b3-e3ff-4224-864f-deb2575ec12c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7569bc7c69-rsrh8\" (UID: \"51e434b3-e3ff-4224-864f-deb2575ec12c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" Jul 10 02:29:46.419583 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.419454 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mtl9\" (UniqueName: \"kubernetes.io/projected/bccd1c52-a5e6-4a92-b128-ed93714208a9-kube-api-access-9mtl9\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:29:46.419583 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.419464 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bccd1c52-a5e6-4a92-b128-ed93714208a9-extensions-socket-volume\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:29:46.419782 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.419760 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51e434b3-e3ff-4224-864f-deb2575ec12c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7569bc7c69-rsrh8\" (UID: \"51e434b3-e3ff-4224-864f-deb2575ec12c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" Jul 10 02:29:46.429688 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.429659 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77h99\" (UniqueName: \"kubernetes.io/projected/51e434b3-e3ff-4224-864f-deb2575ec12c-kube-api-access-77h99\") pod \"kuadrant-operator-controller-manager-7569bc7c69-rsrh8\" (UID: \"51e434b3-e3ff-4224-864f-deb2575ec12c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" Jul 10 02:29:46.538390 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.538288 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" Jul 10 02:29:46.671997 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:46.671967 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8"] Jul 10 02:29:46.673423 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:29:46.673381 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e434b3_e3ff_4224_864f_deb2575ec12c.slice/crio-196f8a0a1b9407ea45bf5d47b264922e75ae99a439f01576f945a6d4ba9062b2 WatchSource:0}: Error finding container 196f8a0a1b9407ea45bf5d47b264922e75ae99a439f01576f945a6d4ba9062b2: Status 404 returned error can't find the container with id 196f8a0a1b9407ea45bf5d47b264922e75ae99a439f01576f945a6d4ba9062b2 Jul 10 02:29:47.131094 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:47.131040 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" event={"ID":"51e434b3-e3ff-4224-864f-deb2575ec12c","Type":"ContainerStarted","Data":"0086173b24a51e30bb28cba97f0c9ad58d2020e49062288941e40c2bfd531acf"} Jul 10 02:29:47.131546 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:47.131114 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" Jul 10 02:29:47.131546 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:47.131135 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" event={"ID":"51e434b3-e3ff-4224-864f-deb2575ec12c","Type":"ContainerStarted","Data":"196f8a0a1b9407ea45bf5d47b264922e75ae99a439f01576f945a6d4ba9062b2"} Jul 10 02:29:47.132531 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:47.132497 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" event={"ID":"bccd1c52-a5e6-4a92-b128-ed93714208a9","Type":"ContainerDied","Data":"e36adfbf05d3e0bb0ff1e3a156adfa04c459799c72084f7f2b0fa517959a4f71"} Jul 10 02:29:47.132531 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:47.132525 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966" Jul 10 02:29:47.132674 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:47.132566 2575 scope.go:117] "RemoveContainer" containerID="93f92915002f3fce93e3d65ca4bc1408c262ac10a73759528fd27ea3a7f173c2" Jul 10 02:29:47.157483 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:47.157427 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" podStartSLOduration=1.157405089 podStartE2EDuration="1.157405089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:29:47.154985534 +0000 UTC m=+416.554438469" watchObservedRunningTime="2026-07-10 02:29:47.157405089 +0000 UTC m=+416.556858003" Jul 10 02:29:47.176806 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:47.176770 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966"] Jul 10 02:29:47.181632 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:47.181607 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-l2966"] Jul 10 02:29:49.129829 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:49.129796 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bccd1c52-a5e6-4a92-b128-ed93714208a9" path="/var/lib/kubelet/pods/bccd1c52-a5e6-4a92-b128-ed93714208a9/volumes" Jul 10 02:29:52.329975 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.329945 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fdcb8fc5-4lw8r"] Jul 10 02:29:52.330432 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.330282 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bccd1c52-a5e6-4a92-b128-ed93714208a9" containerName="manager" Jul 10 02:29:52.330432 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.330294 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bccd1c52-a5e6-4a92-b128-ed93714208a9" containerName="manager" Jul 10 02:29:52.330432 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.330351 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bccd1c52-a5e6-4a92-b128-ed93714208a9" containerName="manager" Jul 10 02:29:52.335760 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.335742 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.341012 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.340984 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 02:29:52.341122 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.341019 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 02:29:52.341122 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.341029 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 02:29:52.341463 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.341443 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 02:29:52.342324 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.342304 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 02:29:52.342440 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.342324 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 02:29:52.342936 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.342726 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5l4mr\"" Jul 10 02:29:52.344846 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.344828 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 02:29:52.345325 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.345304 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 02:29:52.348260 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.348225 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fdcb8fc5-4lw8r"] Jul 10 02:29:52.473939 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.473897 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mstj9\" (UniqueName: \"kubernetes.io/projected/14d72591-381c-4968-a538-76886c0435f2-kube-api-access-mstj9\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.473939 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.473942 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14d72591-381c-4968-a538-76886c0435f2-console-serving-cert\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.474164 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.473961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d72591-381c-4968-a538-76886c0435f2-trusted-ca-bundle\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.474164 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.474071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14d72591-381c-4968-a538-76886c0435f2-oauth-serving-cert\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.474164 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.474131 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14d72591-381c-4968-a538-76886c0435f2-console-oauth-config\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.474305 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.474175 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14d72591-381c-4968-a538-76886c0435f2-console-config\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.474305 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.474205 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14d72591-381c-4968-a538-76886c0435f2-service-ca\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.574706 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.574670 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14d72591-381c-4968-a538-76886c0435f2-console-config\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.574706 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.574713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14d72591-381c-4968-a538-76886c0435f2-service-ca\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.574961 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.574731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mstj9\" (UniqueName: \"kubernetes.io/projected/14d72591-381c-4968-a538-76886c0435f2-kube-api-access-mstj9\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.574961 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.574761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14d72591-381c-4968-a538-76886c0435f2-console-serving-cert\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.574961 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.574781 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d72591-381c-4968-a538-76886c0435f2-trusted-ca-bundle\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.574961 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.574817 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14d72591-381c-4968-a538-76886c0435f2-oauth-serving-cert\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.574961 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.574863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14d72591-381c-4968-a538-76886c0435f2-console-oauth-config\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.575624 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.575597 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14d72591-381c-4968-a538-76886c0435f2-oauth-serving-cert\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.575774 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.575606 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14d72591-381c-4968-a538-76886c0435f2-service-ca\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.575774 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.575763 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14d72591-381c-4968-a538-76886c0435f2-console-config\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.575896 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.575806 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d72591-381c-4968-a538-76886c0435f2-trusted-ca-bundle\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.577309 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.577290 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14d72591-381c-4968-a538-76886c0435f2-console-serving-cert\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.577400 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.577345 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14d72591-381c-4968-a538-76886c0435f2-console-oauth-config\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.586101 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.586054 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mstj9\" (UniqueName: \"kubernetes.io/projected/14d72591-381c-4968-a538-76886c0435f2-kube-api-access-mstj9\") pod \"console-7fdcb8fc5-4lw8r\" (UID: \"14d72591-381c-4968-a538-76886c0435f2\") " pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.645754 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.645733 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:29:52.781709 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:52.781684 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fdcb8fc5-4lw8r"] Jul 10 02:29:52.782860 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:29:52.782831 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d72591_381c_4968_a538_76886c0435f2.slice/crio-c7336d311ee21ff2c265c664ae7698c445ef6e05dbd02475c3952f8ab6ba998d WatchSource:0}: Error finding container c7336d311ee21ff2c265c664ae7698c445ef6e05dbd02475c3952f8ab6ba998d: Status 404 returned error can't find the container with id c7336d311ee21ff2c265c664ae7698c445ef6e05dbd02475c3952f8ab6ba998d Jul 10 02:29:53.167001 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:53.166956 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fdcb8fc5-4lw8r" event={"ID":"14d72591-381c-4968-a538-76886c0435f2","Type":"ContainerStarted","Data":"2db240a126f35992821c60b5f4df8d166bdff3cb151d331355209330668d2d05"} Jul 10 02:29:53.167150 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:53.167031 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fdcb8fc5-4lw8r" event={"ID":"14d72591-381c-4968-a538-76886c0435f2","Type":"ContainerStarted","Data":"c7336d311ee21ff2c265c664ae7698c445ef6e05dbd02475c3952f8ab6ba998d"} Jul 10 02:29:53.189705 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:53.189655 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fdcb8fc5-4lw8r" podStartSLOduration=1.18964211 podStartE2EDuration="1.18964211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:29:53.188400852 +0000 UTC m=+422.587853766" watchObservedRunningTime="2026-07-10 02:29:53.18964211 +0000 UTC m=+422.589095033" Jul 10 02:29:58.141305 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:29:58.141275 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" Jul 10 02:30:02.646884 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:02.646848 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:30:02.647392 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:02.646909 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:30:02.651713 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:02.651692 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:30:03.221200 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:03.221171 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fdcb8fc5-4lw8r" Jul 10 02:30:05.175279 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:05.175228 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:30:05.199721 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:05.199681 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:30:05.199872 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:05.199820 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-mcpr5" Jul 10 02:30:05.202751 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:05.202725 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 02:30:05.212483 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:05.212454 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:30:05.384976 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:05.384937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5ftx\" (UniqueName: \"kubernetes.io/projected/1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2-kube-api-access-t5ftx\") pod \"limitador-limitador-69574b596d-mcpr5\" (UID: \"1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2\") " pod="kuadrant-system/limitador-limitador-69574b596d-mcpr5" Jul 10 02:30:05.385194 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:05.384984 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2-config-file\") pod \"limitador-limitador-69574b596d-mcpr5\" (UID: \"1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2\") " pod="kuadrant-system/limitador-limitador-69574b596d-mcpr5" Jul 10 02:30:05.485444 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:05.485364 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ftx\" (UniqueName: \"kubernetes.io/projected/1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2-kube-api-access-t5ftx\") pod \"limitador-limitador-69574b596d-mcpr5\" (UID: \"1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2\") " pod="kuadrant-system/limitador-limitador-69574b596d-mcpr5" Jul 10 02:30:05.485444 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:05.485416 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2-config-file\") pod \"limitador-limitador-69574b596d-mcpr5\" (UID: \"1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2\") " pod="kuadrant-system/limitador-limitador-69574b596d-mcpr5" Jul 10 02:30:05.486107 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:05.486086 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2-config-file\") pod \"limitador-limitador-69574b596d-mcpr5\" (UID: \"1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2\") " pod="kuadrant-system/limitador-limitador-69574b596d-mcpr5" Jul 10 02:30:05.494471 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:05.494440 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5ftx\" (UniqueName: \"kubernetes.io/projected/1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2-kube-api-access-t5ftx\") pod \"limitador-limitador-69574b596d-mcpr5\" (UID: \"1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2\") " pod="kuadrant-system/limitador-limitador-69574b596d-mcpr5" Jul 10 02:30:05.512418 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:05.512390 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-mcpr5" Jul 10 02:30:05.657358 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:05.657326 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:30:05.658722 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:30:05.658694 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3dfd7c_b199_4dcf_8678_45f81bc3b3b2.slice/crio-36526fd9a8c8bd9b3aab4b65f511b22942c9b23fa3ef74136a7ecd734f645d07 WatchSource:0}: Error finding container 36526fd9a8c8bd9b3aab4b65f511b22942c9b23fa3ef74136a7ecd734f645d07: Status 404 returned error can't find the container with id 36526fd9a8c8bd9b3aab4b65f511b22942c9b23fa3ef74136a7ecd734f645d07 Jul 10 02:30:06.232500 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:06.232462 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-mcpr5" event={"ID":"1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2","Type":"ContainerStarted","Data":"36526fd9a8c8bd9b3aab4b65f511b22942c9b23fa3ef74136a7ecd734f645d07"} Jul 10 02:30:10.257280 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:10.257164 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-mcpr5" event={"ID":"1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2","Type":"ContainerStarted","Data":"3b7ec868b08d0f59c103272d300f4c340be1f949c424f86c24bcf74d3e1d39de"} Jul 10 02:30:10.257686 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:10.257304 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-69574b596d-mcpr5" Jul 10 02:30:10.278275 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:10.278215 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-69574b596d-mcpr5" podStartSLOduration=1.056578355 podStartE2EDuration="5.278201039s" totalImagesPullingTime="4.221622684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:30:10.275947446 +0000 UTC m=+439.675400359" watchObservedRunningTime="2026-07-10 02:30:10.278201039 +0000 UTC m=+439.677653952" Jul 10 02:30:21.262891 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:30:21.262818 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-69574b596d-mcpr5" Jul 10 02:31:20.636119 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:31:20.636082 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:31:46.346403 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:31:46.346325 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:31:56.225750 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:31:56.225714 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:32:18.122229 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:18.122192 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:32:32.635743 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.635706 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz"] Jul 10 02:32:32.639044 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.639020 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.642216 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.642194 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ggkvw\"" Jul 10 02:32:32.642366 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.642199 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 02:32:32.643320 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.643302 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 02:32:32.643421 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.643320 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 02:32:32.645910 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.645822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a121c9bd-daa5-4a09-b851-8639987315b5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.645910 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.645889 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a121c9bd-daa5-4a09-b851-8639987315b5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.646048 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.645916 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzwjc\" (UniqueName: \"kubernetes.io/projected/a121c9bd-daa5-4a09-b851-8639987315b5-kube-api-access-pzwjc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.646048 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.645968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a121c9bd-daa5-4a09-b851-8639987315b5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.646048 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.646004 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a121c9bd-daa5-4a09-b851-8639987315b5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.646163 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.646129 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a121c9bd-daa5-4a09-b851-8639987315b5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.650968 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.650949 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz"] Jul 10 02:32:32.747053 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.746988 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a121c9bd-daa5-4a09-b851-8639987315b5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.747236 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.747086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a121c9bd-daa5-4a09-b851-8639987315b5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.747236 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.747121 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwjc\" (UniqueName: \"kubernetes.io/projected/a121c9bd-daa5-4a09-b851-8639987315b5-kube-api-access-pzwjc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.747236 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.747162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a121c9bd-daa5-4a09-b851-8639987315b5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.747236 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.747202 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a121c9bd-daa5-4a09-b851-8639987315b5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.747462 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.747311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a121c9bd-daa5-4a09-b851-8639987315b5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.750708 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.750673 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a121c9bd-daa5-4a09-b851-8639987315b5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.751025 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.751005 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a121c9bd-daa5-4a09-b851-8639987315b5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.752876 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.752824 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a121c9bd-daa5-4a09-b851-8639987315b5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.753270 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.753226 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a121c9bd-daa5-4a09-b851-8639987315b5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.753580 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.753563 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a121c9bd-daa5-4a09-b851-8639987315b5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.763492 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.763468 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzwjc\" (UniqueName: \"kubernetes.io/projected/a121c9bd-daa5-4a09-b851-8639987315b5-kube-api-access-pzwjc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz\" (UID: \"a121c9bd-daa5-4a09-b851-8639987315b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:32.950271 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:32.950156 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:33.090721 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:33.090697 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz"] Jul 10 02:32:33.092127 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:32:33.092103 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda121c9bd_daa5_4a09_b851_8639987315b5.slice/crio-103421a4227bf7f9b27edda446dfa2d7d015fccdf76b8a39f70870532a8ebbf9 WatchSource:0}: Error finding container 103421a4227bf7f9b27edda446dfa2d7d015fccdf76b8a39f70870532a8ebbf9: Status 404 returned error can't find the container with id 103421a4227bf7f9b27edda446dfa2d7d015fccdf76b8a39f70870532a8ebbf9 Jul 10 02:32:33.980510 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:33.980472 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" event={"ID":"a121c9bd-daa5-4a09-b851-8639987315b5","Type":"ContainerStarted","Data":"103421a4227bf7f9b27edda446dfa2d7d015fccdf76b8a39f70870532a8ebbf9"} Jul 10 02:32:34.043409 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:34.043352 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:32:34.987399 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:34.987361 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" event={"ID":"a121c9bd-daa5-4a09-b851-8639987315b5","Type":"ContainerStarted","Data":"ee3abcfb2ec4d3367d45025bfb51ef8178527fcc9934abb01ea644eea1e5157e"} Jul 10 02:32:35.011957 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:35.011903 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" podStartSLOduration=1.981474719 podStartE2EDuration="3.011883664s" totalImagesPullingTime="1.030408945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:32:35.010234515 +0000 UTC m=+584.409687442" watchObservedRunningTime="2026-07-10 02:32:35.011883664 +0000 UTC m=+584.411336580" Jul 10 02:32:37.935165 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:37.935130 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k"] Jul 10 02:32:37.938634 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:37.938618 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:37.941605 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:37.941583 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 02:32:37.950431 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:37.950409 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k"] Jul 10 02:32:37.997431 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:37.997397 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:37.997607 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:37.997452 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:37.997607 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:37.997477 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:37.997607 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:37.997570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b268b\" (UniqueName: \"kubernetes.io/projected/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-kube-api-access-b268b\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:37.997752 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:37.997633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:37.997816 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:37.997781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:38.098762 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:38.098723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:38.098945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:38.098782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:38.098945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:38.098828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:38.098945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:38.098862 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:38.098945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:38.098889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:38.098945 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:38.098932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b268b\" (UniqueName: \"kubernetes.io/projected/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-kube-api-access-b268b\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:38.099188 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:38.099086 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:38.099188 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:38.099141 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:38.099322 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:38.099214 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:38.101117 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:38.101081 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:38.101391 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:38.101374 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:38.108439 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:38.108403 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b268b\" (UniqueName: \"kubernetes.io/projected/e93b66a8-cb4e-4b21-8b32-bee9f8daacc4-kube-api-access-b268b\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww94k\" (UID: \"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:38.249281 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:38.249180 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:38.384947 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:38.384915 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k"] Jul 10 02:32:38.388733 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:32:38.388709 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93b66a8_cb4e_4b21_8b32_bee9f8daacc4.slice/crio-0ea902783d3739032961d09d6b0ac7f022c4effe867c23ded5562b766a36aad1 WatchSource:0}: Error finding container 0ea902783d3739032961d09d6b0ac7f022c4effe867c23ded5562b766a36aad1: Status 404 returned error can't find the container with id 0ea902783d3739032961d09d6b0ac7f022c4effe867c23ded5562b766a36aad1 Jul 10 02:32:39.008613 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:39.008576 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" event={"ID":"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4","Type":"ContainerStarted","Data":"6d12f2cc69c4491bc1f5e02df9339557a94e9ab91a8e8a92d10b6c7e64f8d2c6"} Jul 10 02:32:39.009083 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:39.008632 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" event={"ID":"e93b66a8-cb4e-4b21-8b32-bee9f8daacc4","Type":"ContainerStarted","Data":"0ea902783d3739032961d09d6b0ac7f022c4effe867c23ded5562b766a36aad1"} Jul 10 02:32:39.030123 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:39.030076 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" podStartSLOduration=1.85136763 podStartE2EDuration="2.030061022s" totalImagesPullingTime="178.693392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:32:39.02838174 +0000 UTC m=+588.427834653" watchObservedRunningTime="2026-07-10 02:32:39.030061022 +0000 UTC m=+588.429513949" Jul 10 02:32:39.530536 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:39.530496 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:32:42.950998 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:42.950949 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:42.951504 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:42.951062 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:42.963800 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:42.963770 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:43.039092 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:43.039062 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz" Jul 10 02:32:48.250024 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:48.249974 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:48.250438 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:48.250045 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:48.262377 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:48.262355 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:49.077445 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:49.077417 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww94k" Jul 10 02:32:51.053723 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:51.053685 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:32:51.054135 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:32:51.053856 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:33:00.641233 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.641191 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h"] Jul 10 02:33:00.646028 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.646002 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.649381 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.649354 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 10 02:33:00.657047 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.657018 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h"] Jul 10 02:33:00.792493 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.792454 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9746\" (UniqueName: \"kubernetes.io/projected/28cc837f-6d23-449e-9762-7f9ce656d3e6-kube-api-access-s9746\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.792657 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.792524 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28cc837f-6d23-449e-9762-7f9ce656d3e6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.792657 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.792589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28cc837f-6d23-449e-9762-7f9ce656d3e6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.792733 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.792651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28cc837f-6d23-449e-9762-7f9ce656d3e6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.792733 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.792690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28cc837f-6d23-449e-9762-7f9ce656d3e6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.792733 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.792706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28cc837f-6d23-449e-9762-7f9ce656d3e6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.893369 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.893284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28cc837f-6d23-449e-9762-7f9ce656d3e6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.893369 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.893326 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28cc837f-6d23-449e-9762-7f9ce656d3e6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.893369 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.893351 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28cc837f-6d23-449e-9762-7f9ce656d3e6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.893369 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.893370 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28cc837f-6d23-449e-9762-7f9ce656d3e6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.893685 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.893410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9746\" (UniqueName: \"kubernetes.io/projected/28cc837f-6d23-449e-9762-7f9ce656d3e6-kube-api-access-s9746\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.893685 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.893473 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28cc837f-6d23-449e-9762-7f9ce656d3e6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.893889 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.893865 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28cc837f-6d23-449e-9762-7f9ce656d3e6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.893956 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.893896 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28cc837f-6d23-449e-9762-7f9ce656d3e6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.893956 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.893924 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28cc837f-6d23-449e-9762-7f9ce656d3e6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.895756 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.895733 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28cc837f-6d23-449e-9762-7f9ce656d3e6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.895941 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.895922 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28cc837f-6d23-449e-9762-7f9ce656d3e6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.903045 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.903025 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9746\" (UniqueName: \"kubernetes.io/projected/28cc837f-6d23-449e-9762-7f9ce656d3e6-kube-api-access-s9746\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h\" (UID: \"28cc837f-6d23-449e-9762-7f9ce656d3e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:00.955327 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:00.955287 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:01.096101 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:01.096072 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h"] Jul 10 02:33:01.097800 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:33:01.097767 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28cc837f_6d23_449e_9762_7f9ce656d3e6.slice/crio-b3ce84b6e1b920dd9ddeefc81565df2723e28a3006fcc58f6dbbb9415eec70f9 WatchSource:0}: Error finding container b3ce84b6e1b920dd9ddeefc81565df2723e28a3006fcc58f6dbbb9415eec70f9: Status 404 returned error can't find the container with id b3ce84b6e1b920dd9ddeefc81565df2723e28a3006fcc58f6dbbb9415eec70f9 Jul 10 02:33:01.131801 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:01.131766 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" event={"ID":"28cc837f-6d23-449e-9762-7f9ce656d3e6","Type":"ContainerStarted","Data":"b3ce84b6e1b920dd9ddeefc81565df2723e28a3006fcc58f6dbbb9415eec70f9"} Jul 10 02:33:02.138865 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:02.138826 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" event={"ID":"28cc837f-6d23-449e-9762-7f9ce656d3e6","Type":"ContainerStarted","Data":"eeb69ec64cb55a7527a989c35068385be8e1fe0c03cae68d6cde5a28ce61685c"} Jul 10 02:33:02.163753 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:02.163688 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" podStartSLOduration=1.993853864 podStartE2EDuration="2.163667195s" totalImagesPullingTime="169.813331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:33:02.158855306 +0000 UTC m=+611.558308219" watchObservedRunningTime="2026-07-10 02:33:02.163667195 +0000 UTC m=+611.563120108" Jul 10 02:33:02.529359 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:02.529278 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:33:10.956398 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:10.956357 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:10.956398 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:10.956406 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:10.968937 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:10.968909 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:33:11.195380 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:33:11.195340 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h" Jul 10 02:34:29.344519 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:34:29.344486 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:34:39.228287 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:34:39.228239 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:34:44.233312 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:34:44.233216 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:34:54.647108 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:34:54.647069 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:34:59.727210 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:34:59.727169 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:35:09.835306 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:35:09.835271 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:35:37.331795 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:35:37.331756 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:35:45.834995 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:35:45.834959 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:36:24.013924 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:36:24.013847 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:36:30.915973 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:36:30.915932 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:36:37.311393 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:36:37.311353 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:36:45.422229 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:36:45.422184 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:36:55.324741 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:36:55.324706 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:37:01.720651 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:37:01.720615 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:37:15.716655 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:37:15.716617 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:37:21.215936 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:37:21.215901 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:37:26.409386 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:37:26.409348 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:37:47.716467 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:37:47.716384 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:37:51.086364 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:37:51.086337 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:37:51.089770 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:37:51.089745 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:37:55.415004 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:37:55.414972 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:38:20.715961 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:38:20.715922 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:38:29.117045 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:38:29.117004 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:38:44.713023 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:38:44.712981 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:38:51.311763 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:38:51.311731 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:39:03.616495 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:03.616458 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:39:10.315814 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:10.315785 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:39:49.185346 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:49.185309 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8"] Jul 10 02:39:49.186115 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:49.185523 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" podUID="51e434b3-e3ff-4224-864f-deb2575ec12c" containerName="manager" containerID="cri-o://0086173b24a51e30bb28cba97f0c9ad58d2020e49062288941e40c2bfd531acf" gracePeriod=10 Jul 10 02:39:51.236232 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.236210 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" Jul 10 02:39:51.314351 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.314320 2575 generic.go:356] "Generic (PLEG): container finished" podID="51e434b3-e3ff-4224-864f-deb2575ec12c" containerID="0086173b24a51e30bb28cba97f0c9ad58d2020e49062288941e40c2bfd531acf" exitCode=0 Jul 10 02:39:51.314505 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.314366 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" Jul 10 02:39:51.314505 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.314379 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" event={"ID":"51e434b3-e3ff-4224-864f-deb2575ec12c","Type":"ContainerDied","Data":"0086173b24a51e30bb28cba97f0c9ad58d2020e49062288941e40c2bfd531acf"} Jul 10 02:39:51.314505 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.314422 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8" event={"ID":"51e434b3-e3ff-4224-864f-deb2575ec12c","Type":"ContainerDied","Data":"196f8a0a1b9407ea45bf5d47b264922e75ae99a439f01576f945a6d4ba9062b2"} Jul 10 02:39:51.314505 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.314444 2575 scope.go:117] "RemoveContainer" containerID="0086173b24a51e30bb28cba97f0c9ad58d2020e49062288941e40c2bfd531acf" Jul 10 02:39:51.327769 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.327745 2575 scope.go:117] "RemoveContainer" containerID="0086173b24a51e30bb28cba97f0c9ad58d2020e49062288941e40c2bfd531acf" Jul 10 02:39:51.328188 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:39:51.328163 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0086173b24a51e30bb28cba97f0c9ad58d2020e49062288941e40c2bfd531acf\": container with ID starting with 0086173b24a51e30bb28cba97f0c9ad58d2020e49062288941e40c2bfd531acf not found: ID does not exist" containerID="0086173b24a51e30bb28cba97f0c9ad58d2020e49062288941e40c2bfd531acf" Jul 10 02:39:51.328358 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.328198 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0086173b24a51e30bb28cba97f0c9ad58d2020e49062288941e40c2bfd531acf"} err="failed to get container status \"0086173b24a51e30bb28cba97f0c9ad58d2020e49062288941e40c2bfd531acf\": rpc error: code = NotFound desc = could not find container \"0086173b24a51e30bb28cba97f0c9ad58d2020e49062288941e40c2bfd531acf\": container with ID starting with 0086173b24a51e30bb28cba97f0c9ad58d2020e49062288941e40c2bfd531acf not found: ID does not exist" Jul 10 02:39:51.386177 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.386159 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51e434b3-e3ff-4224-864f-deb2575ec12c-extensions-socket-volume\") pod \"51e434b3-e3ff-4224-864f-deb2575ec12c\" (UID: \"51e434b3-e3ff-4224-864f-deb2575ec12c\") " Jul 10 02:39:51.386308 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.386220 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77h99\" (UniqueName: \"kubernetes.io/projected/51e434b3-e3ff-4224-864f-deb2575ec12c-kube-api-access-77h99\") pod \"51e434b3-e3ff-4224-864f-deb2575ec12c\" (UID: \"51e434b3-e3ff-4224-864f-deb2575ec12c\") " Jul 10 02:39:51.386541 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.386520 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e434b3-e3ff-4224-864f-deb2575ec12c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "51e434b3-e3ff-4224-864f-deb2575ec12c" (UID: "51e434b3-e3ff-4224-864f-deb2575ec12c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:39:51.388108 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.388089 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e434b3-e3ff-4224-864f-deb2575ec12c-kube-api-access-77h99" (OuterVolumeSpecName: "kube-api-access-77h99") pod "51e434b3-e3ff-4224-864f-deb2575ec12c" (UID: "51e434b3-e3ff-4224-864f-deb2575ec12c"). InnerVolumeSpecName "kube-api-access-77h99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:39:51.487000 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.486966 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51e434b3-e3ff-4224-864f-deb2575ec12c-extensions-socket-volume\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:39:51.487000 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.486995 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77h99\" (UniqueName: \"kubernetes.io/projected/51e434b3-e3ff-4224-864f-deb2575ec12c-kube-api-access-77h99\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:39:51.640452 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.640380 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8"] Jul 10 02:39:51.644077 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:51.644052 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-rsrh8"] Jul 10 02:39:53.129751 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:39:53.129721 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51e434b3-e3ff-4224-864f-deb2575ec12c" path="/var/lib/kubelet/pods/51e434b3-e3ff-4224-864f-deb2575ec12c/volumes" Jul 10 02:40:50.305450 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.305363 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f"] Jul 10 02:40:50.306017 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.305887 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e434b3-e3ff-4224-864f-deb2575ec12c" containerName="manager" Jul 10 02:40:50.306017 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.305907 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e434b3-e3ff-4224-864f-deb2575ec12c" containerName="manager" Jul 10 02:40:50.306132 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.306025 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e434b3-e3ff-4224-864f-deb2575ec12c" containerName="manager" Jul 10 02:40:50.309008 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.308986 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f" Jul 10 02:40:50.312726 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.312706 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x245d\"" Jul 10 02:40:50.324675 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.324644 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f"] Jul 10 02:40:50.402447 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.402417 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km52t\" (UniqueName: \"kubernetes.io/projected/29ef7b85-2eb6-4b3d-8705-b1f76d294e98-kube-api-access-km52t\") pod \"kuadrant-operator-controller-manager-7569bc7c69-cnt7f\" (UID: \"29ef7b85-2eb6-4b3d-8705-b1f76d294e98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f" Jul 10 02:40:50.402604 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.402480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29ef7b85-2eb6-4b3d-8705-b1f76d294e98-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7569bc7c69-cnt7f\" (UID: \"29ef7b85-2eb6-4b3d-8705-b1f76d294e98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f" Jul 10 02:40:50.503466 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.503431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km52t\" (UniqueName: \"kubernetes.io/projected/29ef7b85-2eb6-4b3d-8705-b1f76d294e98-kube-api-access-km52t\") pod \"kuadrant-operator-controller-manager-7569bc7c69-cnt7f\" (UID: \"29ef7b85-2eb6-4b3d-8705-b1f76d294e98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f" Jul 10 02:40:50.503617 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.503495 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29ef7b85-2eb6-4b3d-8705-b1f76d294e98-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7569bc7c69-cnt7f\" (UID: \"29ef7b85-2eb6-4b3d-8705-b1f76d294e98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f" Jul 10 02:40:50.503848 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.503830 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29ef7b85-2eb6-4b3d-8705-b1f76d294e98-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7569bc7c69-cnt7f\" (UID: \"29ef7b85-2eb6-4b3d-8705-b1f76d294e98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f" Jul 10 02:40:50.515797 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.515772 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km52t\" (UniqueName: \"kubernetes.io/projected/29ef7b85-2eb6-4b3d-8705-b1f76d294e98-kube-api-access-km52t\") pod \"kuadrant-operator-controller-manager-7569bc7c69-cnt7f\" (UID: \"29ef7b85-2eb6-4b3d-8705-b1f76d294e98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f" Jul 10 02:40:50.618174 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.618144 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f" Jul 10 02:40:50.763757 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:50.763731 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f"] Jul 10 02:40:50.763906 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:40:50.763884 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ef7b85_2eb6_4b3d_8705_b1f76d294e98.slice/crio-b59c5831c3cb0f26221d7b4225680ef87b232439cbf94af93211d22e5457055a WatchSource:0}: Error finding container b59c5831c3cb0f26221d7b4225680ef87b232439cbf94af93211d22e5457055a: Status 404 returned error can't find the container with id b59c5831c3cb0f26221d7b4225680ef87b232439cbf94af93211d22e5457055a Jul 10 02:40:51.625881 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:51.625853 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f" event={"ID":"29ef7b85-2eb6-4b3d-8705-b1f76d294e98","Type":"ContainerStarted","Data":"adad2f2c3349596dda794ed5d9ea70192382bf509c366673dc221e4d98540f64"} Jul 10 02:40:51.626285 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:51.625901 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f" event={"ID":"29ef7b85-2eb6-4b3d-8705-b1f76d294e98","Type":"ContainerStarted","Data":"b59c5831c3cb0f26221d7b4225680ef87b232439cbf94af93211d22e5457055a"} Jul 10 02:40:51.626285 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:51.625952 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f" Jul 10 02:40:51.649603 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:40:51.649551 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f" podStartSLOduration=1.649537922 podStartE2EDuration="1.649537922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:40:51.648315831 +0000 UTC m=+1081.047768744" watchObservedRunningTime="2026-07-10 02:40:51.649537922 +0000 UTC m=+1081.048990834" Jul 10 02:41:02.633549 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:41:02.633517 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-cnt7f" Jul 10 02:41:09.023136 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:41:09.023101 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:41:17.408869 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:41:17.408834 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:41:30.022572 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:41:30.022527 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:41:36.335106 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:41:36.335070 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:41:43.822312 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:41:43.822260 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:41:49.019173 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:41:49.019137 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:41:54.017763 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:41:54.017722 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:42:03.424742 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:42:03.424708 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:42:08.526290 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:42:08.526254 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:42:18.618964 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:42:18.618886 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:42:23.817150 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:42:23.817107 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:42:34.123278 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:42:34.123227 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:42:40.923902 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:42:40.923866 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:42:50.916635 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:42:50.916598 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:42:51.118676 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:42:51.118647 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:42:51.124193 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:42:51.124172 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:43:16.428745 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:43:16.428711 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:43:22.419684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:43:22.419648 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:43:34.319404 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:43:34.319370 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:43:43.416108 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:43:43.416016 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:44:00.420848 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:44:00.420811 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:44:07.622097 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:44:07.622062 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:44:14.415229 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:44:14.415194 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:44:26.418128 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:44:26.418091 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:44:32.326036 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:44:32.325992 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:44:40.521083 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:44:40.521049 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:44:48.608075 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:44:48.608041 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:44:53.913326 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:44:53.913292 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:45:07.328735 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:45:07.328699 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:45:12.964628 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:45:12.964590 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:45:19.532417 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:45:19.532382 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:45:26.126099 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:45:26.126067 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:45:32.376274 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:45:32.376210 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:45:41.822894 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:45:41.822857 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:45:48.930309 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:45:48.930272 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:45:57.818400 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:45:57.818362 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:46:06.215319 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:46:06.215281 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:46:13.420031 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:46:13.419996 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:46:21.756593 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:46:21.756551 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:46:28.413863 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:46:28.413824 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:46:35.229889 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:46:35.229854 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:46:43.214011 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:46:43.213973 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:46:48.751990 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:46:48.751946 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:47:01.716516 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:47:01.716482 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:47:05.919367 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:47:05.919329 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:47:12.523846 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:47:12.523808 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:47:18.620894 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:47:18.620859 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:47:51.161192 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:47:51.161164 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:47:51.168759 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:47:51.168738 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:49:01.129684 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:49:01.129656 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:49:06.723102 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:49:06.723067 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:49:12.925719 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:49:12.925683 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:49:21.425508 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:49:21.425471 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:49:32.613688 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:49:32.613652 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:50:02.645011 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:02.644938 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:50:26.829896 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:26.829864 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2"] Jul 10 02:50:26.833195 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:26.833178 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:26.835963 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:26.835944 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dc829df0\"/\"openshift-service-ca.crt\"" Jul 10 02:50:26.837135 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:26.837115 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dc829df0\"/\"kube-root-ca.crt\"" Jul 10 02:50:26.837224 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:26.837123 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dc829df0\"/\"e2e-lifecycle-model-dc829df0-kserve-self-signed-certs\"" Jul 10 02:50:26.837340 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:26.837222 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dc829df0\"/\"default-dockercfg-2g7dj\"" Jul 10 02:50:26.844150 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:26.844128 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2"] Jul 10 02:50:27.006222 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.006188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-home\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.006401 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.006267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-tmp-dir\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.006401 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.006318 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n589\" (UniqueName: \"kubernetes.io/projected/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-kube-api-access-7n589\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.006401 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.006346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-model-cache\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.006401 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.006361 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-tls-certs\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.006401 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.006383 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-dshm\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.107170 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.107144 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-model-cache\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.107356 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.107178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-tls-certs\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.107356 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.107204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-dshm\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.107440 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.107417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-home\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.107523 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.107505 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-tmp-dir\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.107595 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.107574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n589\" (UniqueName: \"kubernetes.io/projected/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-kube-api-access-7n589\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.107726 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.107703 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-model-cache\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.107798 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.107733 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-home\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.107851 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.107829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-tmp-dir\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.110068 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.110047 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-dshm\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.110295 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.110281 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-tls-certs\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.116727 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.116705 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n589\" (UniqueName: \"kubernetes.io/projected/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-kube-api-access-7n589\") pod \"e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.142308 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.142286 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:27.276273 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.276225 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2"] Jul 10 02:50:27.276794 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:50:27.276769 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeaf6c8e7_3ea0_48ee_80b2_30df2a6235f1.slice/crio-4ca82a35cbdd662a142f98b5a6227ace2b5ec7ec34304d4b4dd6172107489b1f WatchSource:0}: Error finding container 4ca82a35cbdd662a142f98b5a6227ace2b5ec7ec34304d4b4dd6172107489b1f: Status 404 returned error can't find the container with id 4ca82a35cbdd662a142f98b5a6227ace2b5ec7ec34304d4b4dd6172107489b1f Jul 10 02:50:27.702671 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.702636 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" event={"ID":"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1","Type":"ContainerStarted","Data":"569b33e5f5a6c3675ecaec226f32a91431897bea0217316a1355a9fdeb4e5ac7"} Jul 10 02:50:27.702852 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.702682 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" event={"ID":"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1","Type":"ContainerStarted","Data":"4ca82a35cbdd662a142f98b5a6227ace2b5ec7ec34304d4b4dd6172107489b1f"} Jul 10 02:50:27.721693 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:27.721633 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" podStartSLOduration=1.721616074 podStartE2EDuration="1.721616074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:50:27.720928918 +0000 UTC m=+1657.120381832" watchObservedRunningTime="2026-07-10 02:50:27.721616074 +0000 UTC m=+1657.121068986" Jul 10 02:50:29.621695 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:29.621655 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:50:37.143217 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:37.143179 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:37.143636 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:37.143234 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:37.155942 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:37.155916 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:37.768717 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:37.768688 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:50:45.018388 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:45.018355 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:50:51.613851 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:51.613820 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:50:57.469450 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:57.469418 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2"] Jul 10 02:50:58.821228 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:50:58.821193 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:51:12.490241 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.490191 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" podUID="eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1" containerName="main" containerID="cri-o://569b33e5f5a6c3675ecaec226f32a91431897bea0217316a1355a9fdeb4e5ac7" gracePeriod=45 Jul 10 02:51:12.735590 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.735568 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:51:12.819481 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.819406 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-tmp-dir\") pod \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " Jul 10 02:51:12.819481 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.819440 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-model-cache\") pod \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " Jul 10 02:51:12.819481 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.819474 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-home\") pod \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " Jul 10 02:51:12.819733 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.819496 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-tls-certs\") pod \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " Jul 10 02:51:12.819733 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.819531 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n589\" (UniqueName: \"kubernetes.io/projected/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-kube-api-access-7n589\") pod \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " Jul 10 02:51:12.819733 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.819576 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-dshm\") pod \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\" (UID: \"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1\") " Jul 10 02:51:12.819733 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.819662 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1" (UID: "eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:51:12.819733 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.819706 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1" (UID: "eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:51:12.819980 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.819745 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-home" (OuterVolumeSpecName: "home") pod "eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1" (UID: "eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:51:12.819980 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.819862 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-tmp-dir\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:51:12.819980 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.819874 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-model-cache\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:51:12.819980 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.819883 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-home\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:51:12.821719 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.821695 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-dshm" (OuterVolumeSpecName: "dshm") pod "eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1" (UID: "eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:51:12.821802 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.821773 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-kube-api-access-7n589" (OuterVolumeSpecName: "kube-api-access-7n589") pod "eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1" (UID: "eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1"). InnerVolumeSpecName "kube-api-access-7n589". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:51:12.821865 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.821851 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1" (UID: "eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:51:12.920316 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.920279 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-tls-certs\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:51:12.920316 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.920311 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n589\" (UniqueName: \"kubernetes.io/projected/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-kube-api-access-7n589\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:51:12.920316 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.920324 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1-dshm\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:51:12.941006 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.940978 2575 generic.go:356] "Generic (PLEG): container finished" podID="eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1" containerID="569b33e5f5a6c3675ecaec226f32a91431897bea0217316a1355a9fdeb4e5ac7" exitCode=0 Jul 10 02:51:12.941157 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.941016 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" event={"ID":"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1","Type":"ContainerDied","Data":"569b33e5f5a6c3675ecaec226f32a91431897bea0217316a1355a9fdeb4e5ac7"} Jul 10 02:51:12.941157 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.941033 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" Jul 10 02:51:12.941157 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.941049 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2" event={"ID":"eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1","Type":"ContainerDied","Data":"4ca82a35cbdd662a142f98b5a6227ace2b5ec7ec34304d4b4dd6172107489b1f"} Jul 10 02:51:12.941157 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.941070 2575 scope.go:117] "RemoveContainer" containerID="569b33e5f5a6c3675ecaec226f32a91431897bea0217316a1355a9fdeb4e5ac7" Jul 10 02:51:12.950623 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.950600 2575 scope.go:117] "RemoveContainer" containerID="569b33e5f5a6c3675ecaec226f32a91431897bea0217316a1355a9fdeb4e5ac7" Jul 10 02:51:12.950898 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:51:12.950871 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"569b33e5f5a6c3675ecaec226f32a91431897bea0217316a1355a9fdeb4e5ac7\": container with ID starting with 569b33e5f5a6c3675ecaec226f32a91431897bea0217316a1355a9fdeb4e5ac7 not found: ID does not exist" containerID="569b33e5f5a6c3675ecaec226f32a91431897bea0217316a1355a9fdeb4e5ac7" Jul 10 02:51:12.950950 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.950906 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"569b33e5f5a6c3675ecaec226f32a91431897bea0217316a1355a9fdeb4e5ac7"} err="failed to get container status \"569b33e5f5a6c3675ecaec226f32a91431897bea0217316a1355a9fdeb4e5ac7\": rpc error: code = NotFound desc = could not find container \"569b33e5f5a6c3675ecaec226f32a91431897bea0217316a1355a9fdeb4e5ac7\": container with ID starting with 569b33e5f5a6c3675ecaec226f32a91431897bea0217316a1355a9fdeb4e5ac7 not found: ID does not exist" Jul 10 02:51:12.964374 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.964350 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2"] Jul 10 02:51:12.967875 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:12.967853 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-dc829df0/e2e-lifecycle-model-dc829df0-kserve-855c99dc79-4lps2"] Jul 10 02:51:13.130055 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:13.130017 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1" path="/var/lib/kubelet/pods/eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1/volumes" Jul 10 02:51:36.413344 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:36.413310 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:51:49.226447 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:49.226413 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:51:58.518811 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:51:58.518768 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:52:48.309402 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:52:48.309312 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:52:51.202853 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:52:51.202823 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:52:51.210723 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:52:51.210702 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:52:57.110609 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:52:57.110570 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:53:16.664668 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.664630 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w"] Jul 10 02:53:16.665031 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.664981 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1" containerName="main" Jul 10 02:53:16.665031 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.664993 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1" containerName="main" Jul 10 02:53:16.665102 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.665057 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaf6c8e7-3ea0-48ee-80b2-30df2a6235f1" containerName="main" Jul 10 02:53:16.668209 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.668194 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.671507 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.671489 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-829dac\"/\"openshift-service-ca.crt\"" Jul 10 02:53:16.672675 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.672650 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-829dac\"/\"default-dockercfg-kn9sr\"" Jul 10 02:53:16.672795 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.672714 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-829dac\"/\"test-model-829dac-kserve-self-signed-certs\"" Jul 10 02:53:16.672795 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.672724 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-829dac\"/\"kube-root-ca.crt\"" Jul 10 02:53:16.693061 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.693031 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w"] Jul 10 02:53:16.811020 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.810983 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-model-cache\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.811020 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.811017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-dshm\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.811231 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.811048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-home\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.811231 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.811092 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-tls-certs\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.811231 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.811169 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsmb6\" (UniqueName: \"kubernetes.io/projected/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-kube-api-access-vsmb6\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.811231 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.811208 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-tmp-dir\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.912714 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.912677 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-model-cache\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.912714 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.912715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-dshm\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.912946 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.912751 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-home\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.912946 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.912767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-tls-certs\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.912946 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.912786 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsmb6\" (UniqueName: \"kubernetes.io/projected/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-kube-api-access-vsmb6\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.912946 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.912823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-tmp-dir\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.913167 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.913141 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-model-cache\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.913228 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.913153 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-home\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.913228 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.913217 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-tmp-dir\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.915144 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.915097 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-dshm\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.915364 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.915348 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-tls-certs\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.920555 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.920536 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsmb6\" (UniqueName: \"kubernetes.io/projected/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-kube-api-access-vsmb6\") pod \"test-model-829dac-kserve-595c895b95-qj26w\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:16.977341 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:16.977310 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:17.114355 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:17.114329 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w"] Jul 10 02:53:17.117044 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:53:17.117009 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87f86aa8_bc6b_43f8_8d18_18a2c1ae927e.slice/crio-588e9738974bb769b22d14eb7c477e834f1eed4b50f6e8d3c66654ac4cf22af6 WatchSource:0}: Error finding container 588e9738974bb769b22d14eb7c477e834f1eed4b50f6e8d3c66654ac4cf22af6: Status 404 returned error can't find the container with id 588e9738974bb769b22d14eb7c477e834f1eed4b50f6e8d3c66654ac4cf22af6 Jul 10 02:53:17.581285 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:17.581174 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" event={"ID":"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e","Type":"ContainerStarted","Data":"657df7fad6f6425d2b2a676223c40631b879b52e6a9638dc642cbd5a874447bf"} Jul 10 02:53:17.581285 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:17.581222 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" event={"ID":"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e","Type":"ContainerStarted","Data":"588e9738974bb769b22d14eb7c477e834f1eed4b50f6e8d3c66654ac4cf22af6"} Jul 10 02:53:17.599875 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:17.599828 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" podStartSLOduration=1.599806642 podStartE2EDuration="1.599806642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:53:17.598980989 +0000 UTC m=+1826.998433926" watchObservedRunningTime="2026-07-10 02:53:17.599806642 +0000 UTC m=+1826.999259554" Jul 10 02:53:19.319177 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:19.319137 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:53:26.978060 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:26.978026 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:26.978461 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:26.978078 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:26.990403 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:26.990379 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:27.644137 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:27.644105 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:53:28.211946 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:28.211915 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:53:43.415630 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:43.415588 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:53:53.712462 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:53:53.712425 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:54:01.498169 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:01.498133 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w"] Jul 10 02:54:05.825444 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:05.825404 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:54:16.518903 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.518813 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" podUID="87f86aa8-bc6b-43f8-8d18-18a2c1ae927e" containerName="main" containerID="cri-o://657df7fad6f6425d2b2a676223c40631b879b52e6a9638dc642cbd5a874447bf" gracePeriod=45 Jul 10 02:54:16.765162 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.765139 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:54:16.902441 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.902408 2575 generic.go:356] "Generic (PLEG): container finished" podID="87f86aa8-bc6b-43f8-8d18-18a2c1ae927e" containerID="657df7fad6f6425d2b2a676223c40631b879b52e6a9638dc642cbd5a874447bf" exitCode=0 Jul 10 02:54:16.902607 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.902450 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" Jul 10 02:54:16.902607 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.902493 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" event={"ID":"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e","Type":"ContainerDied","Data":"657df7fad6f6425d2b2a676223c40631b879b52e6a9638dc642cbd5a874447bf"} Jul 10 02:54:16.902607 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.902535 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w" event={"ID":"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e","Type":"ContainerDied","Data":"588e9738974bb769b22d14eb7c477e834f1eed4b50f6e8d3c66654ac4cf22af6"} Jul 10 02:54:16.902607 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.902559 2575 scope.go:117] "RemoveContainer" containerID="657df7fad6f6425d2b2a676223c40631b879b52e6a9638dc642cbd5a874447bf" Jul 10 02:54:16.911636 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.911620 2575 scope.go:117] "RemoveContainer" containerID="657df7fad6f6425d2b2a676223c40631b879b52e6a9638dc642cbd5a874447bf" Jul 10 02:54:16.911897 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:54:16.911877 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657df7fad6f6425d2b2a676223c40631b879b52e6a9638dc642cbd5a874447bf\": container with ID starting with 657df7fad6f6425d2b2a676223c40631b879b52e6a9638dc642cbd5a874447bf not found: ID does not exist" containerID="657df7fad6f6425d2b2a676223c40631b879b52e6a9638dc642cbd5a874447bf" Jul 10 02:54:16.911967 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.911910 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657df7fad6f6425d2b2a676223c40631b879b52e6a9638dc642cbd5a874447bf"} err="failed to get container status \"657df7fad6f6425d2b2a676223c40631b879b52e6a9638dc642cbd5a874447bf\": rpc error: code = NotFound desc = could not find container \"657df7fad6f6425d2b2a676223c40631b879b52e6a9638dc642cbd5a874447bf\": container with ID starting with 657df7fad6f6425d2b2a676223c40631b879b52e6a9638dc642cbd5a874447bf not found: ID does not exist" Jul 10 02:54:16.944859 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.944834 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-tls-certs\") pod \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " Jul 10 02:54:16.944974 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.944897 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-home\") pod \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " Jul 10 02:54:16.944974 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.944925 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsmb6\" (UniqueName: \"kubernetes.io/projected/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-kube-api-access-vsmb6\") pod \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " Jul 10 02:54:16.945087 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.944996 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-model-cache\") pod \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " Jul 10 02:54:16.945087 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.945058 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-tmp-dir\") pod \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " Jul 10 02:54:16.945186 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.945086 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-dshm\") pod \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\" (UID: \"87f86aa8-bc6b-43f8-8d18-18a2c1ae927e\") " Jul 10 02:54:16.945235 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.945196 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-home" (OuterVolumeSpecName: "home") pod "87f86aa8-bc6b-43f8-8d18-18a2c1ae927e" (UID: "87f86aa8-bc6b-43f8-8d18-18a2c1ae927e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:54:16.945368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.945334 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-model-cache" (OuterVolumeSpecName: "model-cache") pod "87f86aa8-bc6b-43f8-8d18-18a2c1ae927e" (UID: "87f86aa8-bc6b-43f8-8d18-18a2c1ae927e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:54:16.945368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.945347 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87f86aa8-bc6b-43f8-8d18-18a2c1ae927e" (UID: "87f86aa8-bc6b-43f8-8d18-18a2c1ae927e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:54:16.945522 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.945393 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-home\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:54:16.947194 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.947171 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-kube-api-access-vsmb6" (OuterVolumeSpecName: "kube-api-access-vsmb6") pod "87f86aa8-bc6b-43f8-8d18-18a2c1ae927e" (UID: "87f86aa8-bc6b-43f8-8d18-18a2c1ae927e"). InnerVolumeSpecName "kube-api-access-vsmb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:54:16.947328 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.947257 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-dshm" (OuterVolumeSpecName: "dshm") pod "87f86aa8-bc6b-43f8-8d18-18a2c1ae927e" (UID: "87f86aa8-bc6b-43f8-8d18-18a2c1ae927e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:54:16.947391 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:16.947375 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87f86aa8-bc6b-43f8-8d18-18a2c1ae927e" (UID: "87f86aa8-bc6b-43f8-8d18-18a2c1ae927e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:54:17.046703 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:17.046668 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-tmp-dir\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:54:17.046703 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:17.046696 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-dshm\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:54:17.046703 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:17.046707 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-tls-certs\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:54:17.046924 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:17.046721 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsmb6\" (UniqueName: \"kubernetes.io/projected/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-kube-api-access-vsmb6\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:54:17.046924 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:17.046732 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e-model-cache\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:54:17.218796 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:17.218712 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w"] Jul 10 02:54:17.222408 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:17.222382 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-829dac/test-model-829dac-kserve-595c895b95-qj26w"] Jul 10 02:54:19.129563 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:19.129530 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87f86aa8-bc6b-43f8-8d18-18a2c1ae927e" path="/var/lib/kubelet/pods/87f86aa8-bc6b-43f8-8d18-18a2c1ae927e/volumes" Jul 10 02:54:48.014538 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:48.014502 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:54:59.220023 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:54:59.219987 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:55:07.210113 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:55:07.210076 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:55:19.410073 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:55:19.410030 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:55:28.211475 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:55:28.211440 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:55:35.710451 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:55:35.710412 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:55:44.515066 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:55:44.514987 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:56:00.624442 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.624405 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d"] Jul 10 02:56:00.624933 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.624779 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87f86aa8-bc6b-43f8-8d18-18a2c1ae927e" containerName="main" Jul 10 02:56:00.624933 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.624791 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="87f86aa8-bc6b-43f8-8d18-18a2c1ae927e" containerName="main" Jul 10 02:56:00.624933 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.624846 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="87f86aa8-bc6b-43f8-8d18-18a2c1ae927e" containerName="main" Jul 10 02:56:00.627865 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.627847 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.630692 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.630665 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1d77d1\"/\"openshift-service-ca.crt\"" Jul 10 02:56:00.631895 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.631876 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1d77d1\"/\"kube-root-ca.crt\"" Jul 10 02:56:00.632035 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.632018 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1d77d1\"/\"auth-test-model-1d77d1-kserve-self-signed-certs\"" Jul 10 02:56:00.632111 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.632050 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1d77d1\"/\"default-dockercfg-jxrrh\"" Jul 10 02:56:00.637287 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.637266 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d"] Jul 10 02:56:00.689359 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.689325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-model-cache\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.689585 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.689547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-home\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.689726 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.689624 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqv6r\" (UniqueName: \"kubernetes.io/projected/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-kube-api-access-bqv6r\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.689726 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.689670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-dshm\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.689726 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.689706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-tls-certs\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.689899 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.689757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-tmp-dir\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.790484 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.790450 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-home\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.790659 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.790491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqv6r\" (UniqueName: \"kubernetes.io/projected/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-kube-api-access-bqv6r\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.790659 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.790526 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-dshm\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.790659 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.790559 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-tls-certs\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.790659 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.790585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-tmp-dir\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.790878 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.790746 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-model-cache\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.790934 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.790913 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-home\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.790971 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.790937 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-tmp-dir\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.791092 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.791075 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-model-cache\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.792829 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.792810 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-dshm\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.793090 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.793072 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-tls-certs\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.798204 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.798181 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqv6r\" (UniqueName: \"kubernetes.io/projected/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-kube-api-access-bqv6r\") pod \"auth-test-model-1d77d1-kserve-6789b9b747-htb4d\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:00.938367 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:00.938228 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:01.068731 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:01.068707 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d"] Jul 10 02:56:01.070135 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:56:01.070109 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b00cf01_05c2_4c57_b3c5_6413f7bc3cf6.slice/crio-2cd0376f07b5cb1a0a31a0a460b93a4eecc464323970e95e8dbf9c7b63f295e0 WatchSource:0}: Error finding container 2cd0376f07b5cb1a0a31a0a460b93a4eecc464323970e95e8dbf9c7b63f295e0: Status 404 returned error can't find the container with id 2cd0376f07b5cb1a0a31a0a460b93a4eecc464323970e95e8dbf9c7b63f295e0 Jul 10 02:56:01.471794 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:01.471762 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" event={"ID":"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6","Type":"ContainerStarted","Data":"f322c9d145ecd277af9f69ad95981fe70696fd7771c10dabbd1ad7de00ffc70a"} Jul 10 02:56:01.471961 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:01.471802 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" event={"ID":"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6","Type":"ContainerStarted","Data":"2cd0376f07b5cb1a0a31a0a460b93a4eecc464323970e95e8dbf9c7b63f295e0"} Jul 10 02:56:01.488313 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:01.488241 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" podStartSLOduration=1.4882270499999999 podStartE2EDuration="1.48822705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:56:01.487084642 +0000 UTC m=+1990.886537569" watchObservedRunningTime="2026-07-10 02:56:01.48822705 +0000 UTC m=+1990.887679962" Jul 10 02:56:03.207694 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:03.207657 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:56:10.939108 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:10.939075 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:10.939556 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:10.939370 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:10.952035 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:10.952009 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:11.541202 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:11.541173 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:56:21.209277 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:21.209234 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:56:38.216186 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:38.216142 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:56:49.709685 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:56:49.709640 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:57:01.212151 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:57:01.212117 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:57:11.220032 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:57:11.220002 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:57:17.713433 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:57:17.713343 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:57:29.211172 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:57:29.211137 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:57:39.418420 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:57:39.418385 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:57:51.236129 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:57:51.236100 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:57:51.244778 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:57:51.244759 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 02:57:51.510118 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:57:51.510039 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:58:00.167411 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:00.167376 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d"] Jul 10 02:58:02.813230 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:02.813198 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf"] Jul 10 02:58:02.816280 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:02.816235 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:02.818915 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:02.818891 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1d77d1\"/\"sub-test-model-1d77d1-kserve-self-signed-certs\"" Jul 10 02:58:02.825729 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:02.825705 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf"] Jul 10 02:58:02.912961 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:02.912923 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:58:02.934888 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:02.934851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-model-cache\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:02.935048 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:02.934905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/201d2a74-dbc4-4b3c-9120-4870ef98245c-tls-certs\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:02.935048 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:02.934987 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-home\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:02.935048 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:02.935015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-dshm\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:02.935153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:02.935064 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-tmp-dir\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:02.935153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:02.935085 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vgf\" (UniqueName: \"kubernetes.io/projected/201d2a74-dbc4-4b3c-9120-4870ef98245c-kube-api-access-p6vgf\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:03.036352 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:03.036317 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-model-cache\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:03.036555 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:03.036373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/201d2a74-dbc4-4b3c-9120-4870ef98245c-tls-certs\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:03.036555 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:03.036405 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-home\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:03.036555 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:03.036419 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-dshm\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:03.036555 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:03.036465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-tmp-dir\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:03.036555 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:03.036493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vgf\" (UniqueName: \"kubernetes.io/projected/201d2a74-dbc4-4b3c-9120-4870ef98245c-kube-api-access-p6vgf\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:03.036851 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:03.036791 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-model-cache\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:03.037097 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:03.037068 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-home\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:03.037279 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:03.037051 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-tmp-dir\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:03.039104 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:03.039079 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-dshm\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:03.039366 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:03.039346 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/201d2a74-dbc4-4b3c-9120-4870ef98245c-tls-certs\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:03.044749 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:03.044723 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vgf\" (UniqueName: \"kubernetes.io/projected/201d2a74-dbc4-4b3c-9120-4870ef98245c-kube-api-access-p6vgf\") pod \"sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:03.130690 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:03.130559 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:03.271896 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:03.271869 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf"] Jul 10 02:58:03.273784 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:58:03.273754 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod201d2a74_dbc4_4b3c_9120_4870ef98245c.slice/crio-d592cb100d27a35add41f48bad5a2c92abac73175e939e9e777489abfbe8b074 WatchSource:0}: Error finding container d592cb100d27a35add41f48bad5a2c92abac73175e939e9e777489abfbe8b074: Status 404 returned error can't find the container with id d592cb100d27a35add41f48bad5a2c92abac73175e939e9e777489abfbe8b074 Jul 10 02:58:04.142705 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:04.142671 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" event={"ID":"201d2a74-dbc4-4b3c-9120-4870ef98245c","Type":"ContainerStarted","Data":"c0e59f253f50403224e79ce0ef2cf95599344ac969ac8ca6c44b52d87f43e809"} Jul 10 02:58:04.143074 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:04.142713 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" event={"ID":"201d2a74-dbc4-4b3c-9120-4870ef98245c","Type":"ContainerStarted","Data":"d592cb100d27a35add41f48bad5a2c92abac73175e939e9e777489abfbe8b074"} Jul 10 02:58:04.158778 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:04.158727 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" podStartSLOduration=2.158707943 podStartE2EDuration="2.158707943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:58:04.157905357 +0000 UTC m=+2113.557358270" watchObservedRunningTime="2026-07-10 02:58:04.158707943 +0000 UTC m=+2113.558160857" Jul 10 02:58:12.118095 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:12.118054 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:58:13.131016 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:13.130987 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:13.131482 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:13.131174 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:13.143370 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:13.143350 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:13.202153 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:13.202133 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:58:15.187642 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.187591 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" podUID="0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6" containerName="main" containerID="cri-o://f322c9d145ecd277af9f69ad95981fe70696fd7771c10dabbd1ad7de00ffc70a" gracePeriod=45 Jul 10 02:58:15.437227 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.437199 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:58:15.549562 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.549488 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-home\") pod \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " Jul 10 02:58:15.549562 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.549546 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqv6r\" (UniqueName: \"kubernetes.io/projected/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-kube-api-access-bqv6r\") pod \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " Jul 10 02:58:15.549752 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.549601 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-dshm\") pod \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " Jul 10 02:58:15.549752 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.549649 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-model-cache\") pod \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " Jul 10 02:58:15.549752 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.549688 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-tls-certs\") pod \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " Jul 10 02:58:15.549752 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.549723 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-tmp-dir\") pod \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\" (UID: \"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6\") " Jul 10 02:58:15.550041 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.549781 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-home" (OuterVolumeSpecName: "home") pod "0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6" (UID: "0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:58:15.550041 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.549925 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6" (UID: "0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:58:15.550041 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.549973 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-home\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:58:15.550041 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.550014 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6" (UID: "0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:58:15.551785 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.551759 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6" (UID: "0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:58:15.551849 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.551808 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-dshm" (OuterVolumeSpecName: "dshm") pod "0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6" (UID: "0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:58:15.551891 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.551866 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-kube-api-access-bqv6r" (OuterVolumeSpecName: "kube-api-access-bqv6r") pod "0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6" (UID: "0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6"). InnerVolumeSpecName "kube-api-access-bqv6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:58:15.650390 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.650367 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqv6r\" (UniqueName: \"kubernetes.io/projected/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-kube-api-access-bqv6r\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:58:15.650390 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.650388 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-dshm\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:58:15.650517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.650397 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-model-cache\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:58:15.650517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.650407 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-tls-certs\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:58:15.650517 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:15.650415 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6-tmp-dir\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:58:16.208838 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:16.208807 2575 generic.go:356] "Generic (PLEG): container finished" podID="0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6" containerID="f322c9d145ecd277af9f69ad95981fe70696fd7771c10dabbd1ad7de00ffc70a" exitCode=0 Jul 10 02:58:16.209320 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:16.208859 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" Jul 10 02:58:16.209320 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:16.208886 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" event={"ID":"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6","Type":"ContainerDied","Data":"f322c9d145ecd277af9f69ad95981fe70696fd7771c10dabbd1ad7de00ffc70a"} Jul 10 02:58:16.209320 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:16.208917 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d" event={"ID":"0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6","Type":"ContainerDied","Data":"2cd0376f07b5cb1a0a31a0a460b93a4eecc464323970e95e8dbf9c7b63f295e0"} Jul 10 02:58:16.209320 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:16.208935 2575 scope.go:117] "RemoveContainer" containerID="f322c9d145ecd277af9f69ad95981fe70696fd7771c10dabbd1ad7de00ffc70a" Jul 10 02:58:16.220526 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:16.220506 2575 scope.go:117] "RemoveContainer" containerID="f322c9d145ecd277af9f69ad95981fe70696fd7771c10dabbd1ad7de00ffc70a" Jul 10 02:58:16.220808 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:58:16.220790 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f322c9d145ecd277af9f69ad95981fe70696fd7771c10dabbd1ad7de00ffc70a\": container with ID starting with f322c9d145ecd277af9f69ad95981fe70696fd7771c10dabbd1ad7de00ffc70a not found: ID does not exist" containerID="f322c9d145ecd277af9f69ad95981fe70696fd7771c10dabbd1ad7de00ffc70a" Jul 10 02:58:16.220876 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:16.220821 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f322c9d145ecd277af9f69ad95981fe70696fd7771c10dabbd1ad7de00ffc70a"} err="failed to get container status \"f322c9d145ecd277af9f69ad95981fe70696fd7771c10dabbd1ad7de00ffc70a\": rpc error: code = NotFound desc = could not find container \"f322c9d145ecd277af9f69ad95981fe70696fd7771c10dabbd1ad7de00ffc70a\": container with ID starting with f322c9d145ecd277af9f69ad95981fe70696fd7771c10dabbd1ad7de00ffc70a not found: ID does not exist" Jul 10 02:58:16.231466 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:16.231443 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d"] Jul 10 02:58:16.233622 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:16.233602 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/auth-test-model-1d77d1-kserve-6789b9b747-htb4d"] Jul 10 02:58:17.130109 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:17.130076 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6" path="/var/lib/kubelet/pods/0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6/volumes" Jul 10 02:58:51.616626 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:58:51.616594 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:59:09.828626 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:09.828594 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:59:23.123130 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:23.123092 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:59:27.717551 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:27.717516 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf"] Jul 10 02:59:34.935716 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:34.935680 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:59:40.681530 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.681501 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj"] Jul 10 02:59:40.681941 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.681880 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6" containerName="main" Jul 10 02:59:40.681941 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.681894 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6" containerName="main" Jul 10 02:59:40.682013 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.681964 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b00cf01-05c2-4c57-b3c5-6413f7bc3cf6" containerName="main" Jul 10 02:59:40.685195 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.685179 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.688227 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.688192 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1d77d1\"/\"rate-test-model-1d77d1-kserve-self-signed-certs\"" Jul 10 02:59:40.704036 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.704007 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj"] Jul 10 02:59:40.808528 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.808494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-model-cache\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.808528 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.808532 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-dshm\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.808758 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.808558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-tmp-dir\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.808758 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.808647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dzvf\" (UniqueName: \"kubernetes.io/projected/70b48950-184a-46b5-8d9e-0044520dd928-kube-api-access-7dzvf\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.809162 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.808726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-home\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.809283 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.809190 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70b48950-184a-46b5-8d9e-0044520dd928-tls-certs\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.910341 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.910310 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-model-cache\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.910507 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.910349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-dshm\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.910507 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.910376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-tmp-dir\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.910507 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.910411 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dzvf\" (UniqueName: \"kubernetes.io/projected/70b48950-184a-46b5-8d9e-0044520dd928-kube-api-access-7dzvf\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.910507 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.910454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-home\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.910507 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.910470 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70b48950-184a-46b5-8d9e-0044520dd928-tls-certs\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.910785 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.910746 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-model-cache\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.910876 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.910851 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-home\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.910876 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.910869 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-tmp-dir\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.912625 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.912604 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-dshm\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.912865 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.912849 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70b48950-184a-46b5-8d9e-0044520dd928-tls-certs\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.920283 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.920239 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dzvf\" (UniqueName: \"kubernetes.io/projected/70b48950-184a-46b5-8d9e-0044520dd928-kube-api-access-7dzvf\") pod \"rate-test-model-1d77d1-kserve-5fc549d77f-tpplj\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:40.995464 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:40.995394 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:41.134444 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:41.134415 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj"] Jul 10 02:59:41.136154 ip-10-0-128-69 kubenswrapper[2575]: W0710 02:59:41.136124 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70b48950_184a_46b5_8d9e_0044520dd928.slice/crio-88ddffba09d09fd9293c7bc81be62e53bd221f2ecaeb80f5c3687dc3e04535db WatchSource:0}: Error finding container 88ddffba09d09fd9293c7bc81be62e53bd221f2ecaeb80f5c3687dc3e04535db: Status 404 returned error can't find the container with id 88ddffba09d09fd9293c7bc81be62e53bd221f2ecaeb80f5c3687dc3e04535db Jul 10 02:59:41.673903 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:41.673864 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" event={"ID":"70b48950-184a-46b5-8d9e-0044520dd928","Type":"ContainerStarted","Data":"8ffff42580fbb259b0127a2b9172aeb1541d25b547a1d948cdca1ccce934d981"} Jul 10 02:59:41.674110 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:41.673916 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" event={"ID":"70b48950-184a-46b5-8d9e-0044520dd928","Type":"ContainerStarted","Data":"88ddffba09d09fd9293c7bc81be62e53bd221f2ecaeb80f5c3687dc3e04535db"} Jul 10 02:59:41.707560 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:41.707511 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" podStartSLOduration=1.707495102 podStartE2EDuration="1.707495102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:59:41.706190153 +0000 UTC m=+2211.105643060" watchObservedRunningTime="2026-07-10 02:59:41.707495102 +0000 UTC m=+2211.106948014" Jul 10 02:59:42.737128 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:42.737083 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" podUID="201d2a74-dbc4-4b3c-9120-4870ef98245c" containerName="main" containerID="cri-o://c0e59f253f50403224e79ce0ef2cf95599344ac969ac8ca6c44b52d87f43e809" gracePeriod=45 Jul 10 02:59:43.004276 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.004230 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:59:43.129066 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.129035 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/201d2a74-dbc4-4b3c-9120-4870ef98245c-tls-certs\") pod \"201d2a74-dbc4-4b3c-9120-4870ef98245c\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " Jul 10 02:59:43.129212 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.129130 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-tmp-dir\") pod \"201d2a74-dbc4-4b3c-9120-4870ef98245c\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " Jul 10 02:59:43.129212 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.129191 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-home\") pod \"201d2a74-dbc4-4b3c-9120-4870ef98245c\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " Jul 10 02:59:43.129325 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.129265 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-dshm\") pod \"201d2a74-dbc4-4b3c-9120-4870ef98245c\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " Jul 10 02:59:43.129325 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.129296 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6vgf\" (UniqueName: \"kubernetes.io/projected/201d2a74-dbc4-4b3c-9120-4870ef98245c-kube-api-access-p6vgf\") pod \"201d2a74-dbc4-4b3c-9120-4870ef98245c\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " Jul 10 02:59:43.129405 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.129345 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-model-cache\") pod \"201d2a74-dbc4-4b3c-9120-4870ef98245c\" (UID: \"201d2a74-dbc4-4b3c-9120-4870ef98245c\") " Jul 10 02:59:43.129506 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.129476 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-home" (OuterVolumeSpecName: "home") pod "201d2a74-dbc4-4b3c-9120-4870ef98245c" (UID: "201d2a74-dbc4-4b3c-9120-4870ef98245c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:59:43.129604 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.129431 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "201d2a74-dbc4-4b3c-9120-4870ef98245c" (UID: "201d2a74-dbc4-4b3c-9120-4870ef98245c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:59:43.129673 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.129606 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-home\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:59:43.129673 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.129653 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-model-cache" (OuterVolumeSpecName: "model-cache") pod "201d2a74-dbc4-4b3c-9120-4870ef98245c" (UID: "201d2a74-dbc4-4b3c-9120-4870ef98245c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:59:43.131223 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.131195 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/201d2a74-dbc4-4b3c-9120-4870ef98245c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "201d2a74-dbc4-4b3c-9120-4870ef98245c" (UID: "201d2a74-dbc4-4b3c-9120-4870ef98245c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:59:43.131368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.131321 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-dshm" (OuterVolumeSpecName: "dshm") pod "201d2a74-dbc4-4b3c-9120-4870ef98245c" (UID: "201d2a74-dbc4-4b3c-9120-4870ef98245c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:59:43.131368 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.131356 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201d2a74-dbc4-4b3c-9120-4870ef98245c-kube-api-access-p6vgf" (OuterVolumeSpecName: "kube-api-access-p6vgf") pod "201d2a74-dbc4-4b3c-9120-4870ef98245c" (UID: "201d2a74-dbc4-4b3c-9120-4870ef98245c"). InnerVolumeSpecName "kube-api-access-p6vgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:59:43.230776 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.230745 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-tmp-dir\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:59:43.230776 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.230772 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-dshm\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:59:43.230951 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.230784 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6vgf\" (UniqueName: \"kubernetes.io/projected/201d2a74-dbc4-4b3c-9120-4870ef98245c-kube-api-access-p6vgf\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:59:43.230951 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.230798 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/201d2a74-dbc4-4b3c-9120-4870ef98245c-model-cache\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:59:43.230951 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.230810 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/201d2a74-dbc4-4b3c-9120-4870ef98245c-tls-certs\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 02:59:43.471881 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.471840 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 02:59:43.685810 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.685778 2575 generic.go:356] "Generic (PLEG): container finished" podID="201d2a74-dbc4-4b3c-9120-4870ef98245c" containerID="c0e59f253f50403224e79ce0ef2cf95599344ac969ac8ca6c44b52d87f43e809" exitCode=0 Jul 10 02:59:43.685995 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.685816 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" event={"ID":"201d2a74-dbc4-4b3c-9120-4870ef98245c","Type":"ContainerDied","Data":"c0e59f253f50403224e79ce0ef2cf95599344ac969ac8ca6c44b52d87f43e809"} Jul 10 02:59:43.685995 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.685820 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" Jul 10 02:59:43.685995 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.685856 2575 scope.go:117] "RemoveContainer" containerID="c0e59f253f50403224e79ce0ef2cf95599344ac969ac8ca6c44b52d87f43e809" Jul 10 02:59:43.685995 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.685845 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf" event={"ID":"201d2a74-dbc4-4b3c-9120-4870ef98245c","Type":"ContainerDied","Data":"d592cb100d27a35add41f48bad5a2c92abac73175e939e9e777489abfbe8b074"} Jul 10 02:59:43.694905 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.694886 2575 scope.go:117] "RemoveContainer" containerID="c0e59f253f50403224e79ce0ef2cf95599344ac969ac8ca6c44b52d87f43e809" Jul 10 02:59:43.695164 ip-10-0-128-69 kubenswrapper[2575]: E0710 02:59:43.695146 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e59f253f50403224e79ce0ef2cf95599344ac969ac8ca6c44b52d87f43e809\": container with ID starting with c0e59f253f50403224e79ce0ef2cf95599344ac969ac8ca6c44b52d87f43e809 not found: ID does not exist" containerID="c0e59f253f50403224e79ce0ef2cf95599344ac969ac8ca6c44b52d87f43e809" Jul 10 02:59:43.695215 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.695172 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e59f253f50403224e79ce0ef2cf95599344ac969ac8ca6c44b52d87f43e809"} err="failed to get container status \"c0e59f253f50403224e79ce0ef2cf95599344ac969ac8ca6c44b52d87f43e809\": rpc error: code = NotFound desc = could not find container \"c0e59f253f50403224e79ce0ef2cf95599344ac969ac8ca6c44b52d87f43e809\": container with ID starting with c0e59f253f50403224e79ce0ef2cf95599344ac969ac8ca6c44b52d87f43e809 not found: ID does not exist" Jul 10 02:59:43.714229 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.714202 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf"] Jul 10 02:59:43.723263 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:43.723197 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/sub-test-model-1d77d1-kserve-5c5bdbc999-mzhbf"] Jul 10 02:59:45.129974 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:45.129943 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="201d2a74-dbc4-4b3c-9120-4870ef98245c" path="/var/lib/kubelet/pods/201d2a74-dbc4-4b3c-9120-4870ef98245c/volumes" Jul 10 02:59:50.996215 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:50.996182 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:50.996215 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:50.996225 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:51.008570 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:51.008543 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 02:59:51.746032 ip-10-0-128-69 kubenswrapper[2575]: I0710 02:59:51.745998 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 03:00:15.408834 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:15.408753 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 03:00:32.311169 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:32.311135 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 03:00:40.818266 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:40.818209 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 03:00:46.301616 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:46.301585 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj"] Jul 10 03:00:51.469051 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:51.469013 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7371-139815049209776"] Jul 10 03:00:51.469593 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:51.469573 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201d2a74-dbc4-4b3c-9120-4870ef98245c" containerName="main" Jul 10 03:00:51.469702 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:51.469595 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="201d2a74-dbc4-4b3c-9120-4870ef98245c" containerName="main" Jul 10 03:00:51.469758 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:51.469728 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="201d2a74-dbc4-4b3c-9120-4870ef98245c" containerName="main" Jul 10 03:00:51.472894 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:51.472874 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815049209776" Jul 10 03:00:51.475438 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:51.475418 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-ff2w2\"" Jul 10 03:00:51.479961 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:51.479939 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7371-139815049209776"] Jul 10 03:00:51.635864 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:51.635827 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxdzp\" (UniqueName: \"kubernetes.io/projected/289ed30e-84a4-4d1d-83d5-ce0bc3436e84-kube-api-access-jxdzp\") pod \"test-curl-7371-139815049209776\" (UID: \"289ed30e-84a4-4d1d-83d5-ce0bc3436e84\") " pod="openshift-ingress/test-curl-7371-139815049209776" Jul 10 03:00:51.737462 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:51.737377 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxdzp\" (UniqueName: \"kubernetes.io/projected/289ed30e-84a4-4d1d-83d5-ce0bc3436e84-kube-api-access-jxdzp\") pod \"test-curl-7371-139815049209776\" (UID: \"289ed30e-84a4-4d1d-83d5-ce0bc3436e84\") " pod="openshift-ingress/test-curl-7371-139815049209776" Jul 10 03:00:51.745797 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:51.745770 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxdzp\" (UniqueName: \"kubernetes.io/projected/289ed30e-84a4-4d1d-83d5-ce0bc3436e84-kube-api-access-jxdzp\") pod \"test-curl-7371-139815049209776\" (UID: \"289ed30e-84a4-4d1d-83d5-ce0bc3436e84\") " pod="openshift-ingress/test-curl-7371-139815049209776" Jul 10 03:00:51.783962 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:51.783930 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815049209776" Jul 10 03:00:51.921121 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:51.921095 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7371-139815049209776"] Jul 10 03:00:51.923174 ip-10-0-128-69 kubenswrapper[2575]: W0710 03:00:51.923152 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod289ed30e_84a4_4d1d_83d5_ce0bc3436e84.slice/crio-67775a4ec3992d9161819be035d1b60da3fb54cc9d5445af74c9bd00c0203b95 WatchSource:0}: Error finding container 67775a4ec3992d9161819be035d1b60da3fb54cc9d5445af74c9bd00c0203b95: Status 404 returned error can't find the container with id 67775a4ec3992d9161819be035d1b60da3fb54cc9d5445af74c9bd00c0203b95 Jul 10 03:00:51.924543 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:51.924523 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 03:00:52.054927 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:52.054839 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7371-139815049209776" event={"ID":"289ed30e-84a4-4d1d-83d5-ce0bc3436e84","Type":"ContainerStarted","Data":"67775a4ec3992d9161819be035d1b60da3fb54cc9d5445af74c9bd00c0203b95"} Jul 10 03:00:53.063352 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:53.063234 2575 generic.go:356] "Generic (PLEG): container finished" podID="289ed30e-84a4-4d1d-83d5-ce0bc3436e84" containerID="d98deee221ddc309b59783cfea526017913e7f8d5e231188ccb25ae915aea89e" exitCode=0 Jul 10 03:00:53.063352 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:53.063321 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7371-139815049209776" event={"ID":"289ed30e-84a4-4d1d-83d5-ce0bc3436e84","Type":"ContainerDied","Data":"d98deee221ddc309b59783cfea526017913e7f8d5e231188ccb25ae915aea89e"} Jul 10 03:00:53.312110 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:53.312073 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 03:00:54.211553 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.211529 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815049209776" Jul 10 03:00:54.248918 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.248895 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7371-139815049209776_289ed30e-84a4-4d1d-83d5-ce0bc3436e84/test-curl-7371-139815049209776/0.log" Jul 10 03:00:54.256194 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.256172 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxdzp\" (UniqueName: \"kubernetes.io/projected/289ed30e-84a4-4d1d-83d5-ce0bc3436e84-kube-api-access-jxdzp\") pod \"289ed30e-84a4-4d1d-83d5-ce0bc3436e84\" (UID: \"289ed30e-84a4-4d1d-83d5-ce0bc3436e84\") " Jul 10 03:00:54.258207 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.258184 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289ed30e-84a4-4d1d-83d5-ce0bc3436e84-kube-api-access-jxdzp" (OuterVolumeSpecName: "kube-api-access-jxdzp") pod "289ed30e-84a4-4d1d-83d5-ce0bc3436e84" (UID: "289ed30e-84a4-4d1d-83d5-ce0bc3436e84"). InnerVolumeSpecName "kube-api-access-jxdzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:00:54.274970 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.274940 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7371-139815049209776"] Jul 10 03:00:54.279061 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.279040 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7371-139815049209776"] Jul 10 03:00:54.357215 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.357187 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxdzp\" (UniqueName: \"kubernetes.io/projected/289ed30e-84a4-4d1d-83d5-ce0bc3436e84-kube-api-access-jxdzp\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 03:00:54.429395 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.429362 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7371-139815048300416"] Jul 10 03:00:54.429735 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.429722 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289ed30e-84a4-4d1d-83d5-ce0bc3436e84" containerName="test-curl-7371-139815049209776" Jul 10 03:00:54.429782 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.429736 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="289ed30e-84a4-4d1d-83d5-ce0bc3436e84" containerName="test-curl-7371-139815049209776" Jul 10 03:00:54.429818 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.429799 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="289ed30e-84a4-4d1d-83d5-ce0bc3436e84" containerName="test-curl-7371-139815049209776" Jul 10 03:00:54.432678 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.432661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:00:54.438326 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.438301 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7371-139815048300416"] Jul 10 03:00:54.458335 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.458310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbl7s\" (UniqueName: \"kubernetes.io/projected/c317093d-8136-46b2-8bf7-7ce3bd8a15fe-kube-api-access-xbl7s\") pod \"test-curl-7371-139815048300416\" (UID: \"c317093d-8136-46b2-8bf7-7ce3bd8a15fe\") " pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:00:54.559299 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.559265 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbl7s\" (UniqueName: \"kubernetes.io/projected/c317093d-8136-46b2-8bf7-7ce3bd8a15fe-kube-api-access-xbl7s\") pod \"test-curl-7371-139815048300416\" (UID: \"c317093d-8136-46b2-8bf7-7ce3bd8a15fe\") " pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:00:54.568865 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.568835 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbl7s\" (UniqueName: \"kubernetes.io/projected/c317093d-8136-46b2-8bf7-7ce3bd8a15fe-kube-api-access-xbl7s\") pod \"test-curl-7371-139815048300416\" (UID: \"c317093d-8136-46b2-8bf7-7ce3bd8a15fe\") " pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:00:54.743480 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.743398 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:00:54.877948 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:54.877920 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7371-139815048300416"] Jul 10 03:00:54.880220 ip-10-0-128-69 kubenswrapper[2575]: W0710 03:00:54.880193 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc317093d_8136_46b2_8bf7_7ce3bd8a15fe.slice/crio-490ea235c960108326faf3838da84af2c2f22deb5df859333e7d9aded7bc0cc5 WatchSource:0}: Error finding container 490ea235c960108326faf3838da84af2c2f22deb5df859333e7d9aded7bc0cc5: Status 404 returned error can't find the container with id 490ea235c960108326faf3838da84af2c2f22deb5df859333e7d9aded7bc0cc5 Jul 10 03:00:55.075461 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:55.075363 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7371-139815048300416" event={"ID":"c317093d-8136-46b2-8bf7-7ce3bd8a15fe","Type":"ContainerStarted","Data":"490ea235c960108326faf3838da84af2c2f22deb5df859333e7d9aded7bc0cc5"} Jul 10 03:00:55.076576 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:55.076553 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67775a4ec3992d9161819be035d1b60da3fb54cc9d5445af74c9bd00c0203b95" Jul 10 03:00:55.076576 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:55.076562 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815049209776" Jul 10 03:00:55.087857 ip-10-0-128-69 kubenswrapper[2575]: E0710 03:00:55.087819 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7371-139815049209776\" is forbidden: User \"system:node:ip-10-0-128-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-69.ec2.internal' and this object" podUID="289ed30e-84a4-4d1d-83d5-ce0bc3436e84" pod="openshift-ingress/test-curl-7371-139815049209776" Jul 10 03:00:55.130001 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:55.129975 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="289ed30e-84a4-4d1d-83d5-ce0bc3436e84" path="/var/lib/kubelet/pods/289ed30e-84a4-4d1d-83d5-ce0bc3436e84/volumes" Jul 10 03:00:56.084077 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:56.084046 2575 generic.go:356] "Generic (PLEG): container finished" podID="c317093d-8136-46b2-8bf7-7ce3bd8a15fe" containerID="6effbcf2a07da676796400941776e20180799ee0ef8db4c16277c11f8e86ef6f" exitCode=0 Jul 10 03:00:56.084519 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:56.084139 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7371-139815048300416" event={"ID":"c317093d-8136-46b2-8bf7-7ce3bd8a15fe","Type":"ContainerDied","Data":"6effbcf2a07da676796400941776e20180799ee0ef8db4c16277c11f8e86ef6f"} Jul 10 03:00:57.232775 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.232747 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:00:57.252829 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.252802 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7371-139815048300416_c317093d-8136-46b2-8bf7-7ce3bd8a15fe/test-curl-7371-139815048300416/0.log" Jul 10 03:00:57.275999 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.275967 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7371-139815048300416"] Jul 10 03:00:57.279183 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.279158 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7371-139815048300416"] Jul 10 03:00:57.280860 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.280846 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbl7s\" (UniqueName: \"kubernetes.io/projected/c317093d-8136-46b2-8bf7-7ce3bd8a15fe-kube-api-access-xbl7s\") pod \"c317093d-8136-46b2-8bf7-7ce3bd8a15fe\" (UID: \"c317093d-8136-46b2-8bf7-7ce3bd8a15fe\") " Jul 10 03:00:57.282969 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.282941 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c317093d-8136-46b2-8bf7-7ce3bd8a15fe-kube-api-access-xbl7s" (OuterVolumeSpecName: "kube-api-access-xbl7s") pod "c317093d-8136-46b2-8bf7-7ce3bd8a15fe" (UID: "c317093d-8136-46b2-8bf7-7ce3bd8a15fe"). InnerVolumeSpecName "kube-api-access-xbl7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:00:57.381595 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.381552 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbl7s\" (UniqueName: \"kubernetes.io/projected/c317093d-8136-46b2-8bf7-7ce3bd8a15fe-kube-api-access-xbl7s\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 03:00:57.406851 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.406819 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7371-139815048300416"] Jul 10 03:00:57.407224 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.407212 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c317093d-8136-46b2-8bf7-7ce3bd8a15fe" containerName="test-curl-7371-139815048300416" Jul 10 03:00:57.407283 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.407226 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c317093d-8136-46b2-8bf7-7ce3bd8a15fe" containerName="test-curl-7371-139815048300416" Jul 10 03:00:57.407325 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.407305 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c317093d-8136-46b2-8bf7-7ce3bd8a15fe" containerName="test-curl-7371-139815048300416" Jul 10 03:00:57.410060 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.410045 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:00:57.416372 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.416346 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7371-139815048300416"] Jul 10 03:00:57.482654 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.482624 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvbd\" (UniqueName: \"kubernetes.io/projected/c56219f4-590b-4b0a-be6e-5291b0296f03-kube-api-access-wcvbd\") pod \"test-curl-7371-139815048300416\" (UID: \"c56219f4-590b-4b0a-be6e-5291b0296f03\") " pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:00:57.583526 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.583491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvbd\" (UniqueName: \"kubernetes.io/projected/c56219f4-590b-4b0a-be6e-5291b0296f03-kube-api-access-wcvbd\") pod \"test-curl-7371-139815048300416\" (UID: \"c56219f4-590b-4b0a-be6e-5291b0296f03\") " pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:00:57.592067 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.592041 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvbd\" (UniqueName: \"kubernetes.io/projected/c56219f4-590b-4b0a-be6e-5291b0296f03-kube-api-access-wcvbd\") pod \"test-curl-7371-139815048300416\" (UID: \"c56219f4-590b-4b0a-be6e-5291b0296f03\") " pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:00:57.720302 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.720213 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:00:57.850410 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:57.850388 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7371-139815048300416"] Jul 10 03:00:57.851920 ip-10-0-128-69 kubenswrapper[2575]: W0710 03:00:57.851894 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc56219f4_590b_4b0a_be6e_5291b0296f03.slice/crio-bb103b2f07b524f481e1a1881fc6b8818b4354be799d44a3910bca264c2281de WatchSource:0}: Error finding container bb103b2f07b524f481e1a1881fc6b8818b4354be799d44a3910bca264c2281de: Status 404 returned error can't find the container with id bb103b2f07b524f481e1a1881fc6b8818b4354be799d44a3910bca264c2281de Jul 10 03:00:58.095759 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:58.095678 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:00:58.095759 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:58.095694 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="490ea235c960108326faf3838da84af2c2f22deb5df859333e7d9aded7bc0cc5" Jul 10 03:00:58.096944 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:58.096920 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7371-139815048300416" event={"ID":"c56219f4-590b-4b0a-be6e-5291b0296f03","Type":"ContainerStarted","Data":"bb103b2f07b524f481e1a1881fc6b8818b4354be799d44a3910bca264c2281de"} Jul 10 03:00:58.107815 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:58.107787 2575 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7371-139815048300416" oldPodUID="c317093d-8136-46b2-8bf7-7ce3bd8a15fe" podUID="c56219f4-590b-4b0a-be6e-5291b0296f03" Jul 10 03:00:59.106793 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:59.106760 2575 generic.go:356] "Generic (PLEG): container finished" podID="c56219f4-590b-4b0a-be6e-5291b0296f03" containerID="58052109b1df76b7df852e21b4b6b0a7871a3f6d2dfbb941562d0badd43c753d" exitCode=0 Jul 10 03:00:59.107160 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:59.106797 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7371-139815048300416" event={"ID":"c56219f4-590b-4b0a-be6e-5291b0296f03","Type":"ContainerDied","Data":"58052109b1df76b7df852e21b4b6b0a7871a3f6d2dfbb941562d0badd43c753d"} Jul 10 03:00:59.128827 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:00:59.128800 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c317093d-8136-46b2-8bf7-7ce3bd8a15fe" path="/var/lib/kubelet/pods/c317093d-8136-46b2-8bf7-7ce3bd8a15fe/volumes" Jul 10 03:01:00.250985 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.250958 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:01:00.270825 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.270801 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7371-139815048300416_c56219f4-590b-4b0a-be6e-5291b0296f03/test-curl-7371-139815048300416/0.log" Jul 10 03:01:00.295261 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.295220 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7371-139815048300416"] Jul 10 03:01:00.298791 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.298765 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7371-139815048300416"] Jul 10 03:01:00.305301 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.305282 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcvbd\" (UniqueName: \"kubernetes.io/projected/c56219f4-590b-4b0a-be6e-5291b0296f03-kube-api-access-wcvbd\") pod \"c56219f4-590b-4b0a-be6e-5291b0296f03\" (UID: \"c56219f4-590b-4b0a-be6e-5291b0296f03\") " Jul 10 03:01:00.307290 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.307268 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56219f4-590b-4b0a-be6e-5291b0296f03-kube-api-access-wcvbd" (OuterVolumeSpecName: "kube-api-access-wcvbd") pod "c56219f4-590b-4b0a-be6e-5291b0296f03" (UID: "c56219f4-590b-4b0a-be6e-5291b0296f03"). InnerVolumeSpecName "kube-api-access-wcvbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:01:00.406402 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.406327 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvbd\" (UniqueName: \"kubernetes.io/projected/c56219f4-590b-4b0a-be6e-5291b0296f03-kube-api-access-wcvbd\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 03:01:00.413840 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.413816 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7371-139815048301424"] Jul 10 03:01:00.414186 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.414173 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c56219f4-590b-4b0a-be6e-5291b0296f03" containerName="test-curl-7371-139815048300416" Jul 10 03:01:00.414240 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.414188 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c56219f4-590b-4b0a-be6e-5291b0296f03" containerName="test-curl-7371-139815048300416" Jul 10 03:01:00.414296 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.414266 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c56219f4-590b-4b0a-be6e-5291b0296f03" containerName="test-curl-7371-139815048300416" Jul 10 03:01:00.417129 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.417114 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048301424" Jul 10 03:01:00.423067 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.423041 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7371-139815048301424"] Jul 10 03:01:00.507150 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.507118 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdls6\" (UniqueName: \"kubernetes.io/projected/99390220-a0ab-45fd-b548-47befe758cb1-kube-api-access-rdls6\") pod \"test-curl-7371-139815048301424\" (UID: \"99390220-a0ab-45fd-b548-47befe758cb1\") " pod="openshift-ingress/test-curl-7371-139815048301424" Jul 10 03:01:00.608347 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.608313 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdls6\" (UniqueName: \"kubernetes.io/projected/99390220-a0ab-45fd-b548-47befe758cb1-kube-api-access-rdls6\") pod \"test-curl-7371-139815048301424\" (UID: \"99390220-a0ab-45fd-b548-47befe758cb1\") " pod="openshift-ingress/test-curl-7371-139815048301424" Jul 10 03:01:00.617581 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.617560 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdls6\" (UniqueName: \"kubernetes.io/projected/99390220-a0ab-45fd-b548-47befe758cb1-kube-api-access-rdls6\") pod \"test-curl-7371-139815048301424\" (UID: \"99390220-a0ab-45fd-b548-47befe758cb1\") " pod="openshift-ingress/test-curl-7371-139815048301424" Jul 10 03:01:00.727042 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.726967 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048301424" Jul 10 03:01:00.859307 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:00.859282 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7371-139815048301424"] Jul 10 03:01:00.860989 ip-10-0-128-69 kubenswrapper[2575]: W0710 03:01:00.860965 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99390220_a0ab_45fd_b548_47befe758cb1.slice/crio-8291e6ba7af78eb1e256dbd35db2cd632714f582c102c5407bc7afb665aea654 WatchSource:0}: Error finding container 8291e6ba7af78eb1e256dbd35db2cd632714f582c102c5407bc7afb665aea654: Status 404 returned error can't find the container with id 8291e6ba7af78eb1e256dbd35db2cd632714f582c102c5407bc7afb665aea654 Jul 10 03:01:01.118181 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.118150 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7371-139815048301424" event={"ID":"99390220-a0ab-45fd-b548-47befe758cb1","Type":"ContainerStarted","Data":"8291e6ba7af78eb1e256dbd35db2cd632714f582c102c5407bc7afb665aea654"} Jul 10 03:01:01.119308 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.119288 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:01:01.119396 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.119315 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb103b2f07b524f481e1a1881fc6b8818b4354be799d44a3910bca264c2281de" Jul 10 03:01:01.131661 ip-10-0-128-69 kubenswrapper[2575]: E0710 03:01:01.131623 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7371-139815048300416\" is forbidden: User \"system:node:ip-10-0-128-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-69.ec2.internal' and this object" podUID="c56219f4-590b-4b0a-be6e-5291b0296f03" pod="openshift-ingress/test-curl-7371-139815048300416" Jul 10 03:01:01.136565 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.136541 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c56219f4-590b-4b0a-be6e-5291b0296f03" path="/var/lib/kubelet/pods/c56219f4-590b-4b0a-be6e-5291b0296f03/volumes" Jul 10 03:01:01.321423 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.321361 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" podUID="70b48950-184a-46b5-8d9e-0044520dd928" containerName="main" containerID="cri-o://8ffff42580fbb259b0127a2b9172aeb1541d25b547a1d948cdca1ccce934d981" gracePeriod=45 Jul 10 03:01:01.601240 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.601210 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 03:01:01.617155 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.616962 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dzvf\" (UniqueName: \"kubernetes.io/projected/70b48950-184a-46b5-8d9e-0044520dd928-kube-api-access-7dzvf\") pod \"70b48950-184a-46b5-8d9e-0044520dd928\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " Jul 10 03:01:01.617155 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.617028 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70b48950-184a-46b5-8d9e-0044520dd928-tls-certs\") pod \"70b48950-184a-46b5-8d9e-0044520dd928\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " Jul 10 03:01:01.617155 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.617072 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-model-cache\") pod \"70b48950-184a-46b5-8d9e-0044520dd928\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " Jul 10 03:01:01.617155 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.617095 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-dshm\") pod \"70b48950-184a-46b5-8d9e-0044520dd928\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " Jul 10 03:01:01.617155 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.617132 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-home\") pod \"70b48950-184a-46b5-8d9e-0044520dd928\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " Jul 10 03:01:01.617584 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.617184 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-tmp-dir\") pod \"70b48950-184a-46b5-8d9e-0044520dd928\" (UID: \"70b48950-184a-46b5-8d9e-0044520dd928\") " Jul 10 03:01:01.617715 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.617687 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70b48950-184a-46b5-8d9e-0044520dd928" (UID: "70b48950-184a-46b5-8d9e-0044520dd928"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 03:01:01.617977 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.617951 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-model-cache" (OuterVolumeSpecName: "model-cache") pod "70b48950-184a-46b5-8d9e-0044520dd928" (UID: "70b48950-184a-46b5-8d9e-0044520dd928"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 03:01:01.618187 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.618161 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-home" (OuterVolumeSpecName: "home") pod "70b48950-184a-46b5-8d9e-0044520dd928" (UID: "70b48950-184a-46b5-8d9e-0044520dd928"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 03:01:01.619664 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.619620 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b48950-184a-46b5-8d9e-0044520dd928-kube-api-access-7dzvf" (OuterVolumeSpecName: "kube-api-access-7dzvf") pod "70b48950-184a-46b5-8d9e-0044520dd928" (UID: "70b48950-184a-46b5-8d9e-0044520dd928"). InnerVolumeSpecName "kube-api-access-7dzvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:01:01.620193 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.620169 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-dshm" (OuterVolumeSpecName: "dshm") pod "70b48950-184a-46b5-8d9e-0044520dd928" (UID: "70b48950-184a-46b5-8d9e-0044520dd928"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 03:01:01.620286 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.620267 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b48950-184a-46b5-8d9e-0044520dd928-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70b48950-184a-46b5-8d9e-0044520dd928" (UID: "70b48950-184a-46b5-8d9e-0044520dd928"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 03:01:01.718273 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.718172 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dzvf\" (UniqueName: \"kubernetes.io/projected/70b48950-184a-46b5-8d9e-0044520dd928-kube-api-access-7dzvf\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 03:01:01.718273 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.718213 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70b48950-184a-46b5-8d9e-0044520dd928-tls-certs\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 03:01:01.718273 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.718225 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-model-cache\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 03:01:01.718273 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.718233 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-dshm\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 03:01:01.718273 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.718255 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-home\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 03:01:01.718548 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:01.718290 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b48950-184a-46b5-8d9e-0044520dd928-tmp-dir\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 03:01:02.127089 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:02.127053 2575 generic.go:356] "Generic (PLEG): container finished" podID="70b48950-184a-46b5-8d9e-0044520dd928" containerID="8ffff42580fbb259b0127a2b9172aeb1541d25b547a1d948cdca1ccce934d981" exitCode=0 Jul 10 03:01:02.127089 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:02.127088 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" Jul 10 03:01:02.127350 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:02.127135 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" event={"ID":"70b48950-184a-46b5-8d9e-0044520dd928","Type":"ContainerDied","Data":"8ffff42580fbb259b0127a2b9172aeb1541d25b547a1d948cdca1ccce934d981"} Jul 10 03:01:02.127350 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:02.127173 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj" event={"ID":"70b48950-184a-46b5-8d9e-0044520dd928","Type":"ContainerDied","Data":"88ddffba09d09fd9293c7bc81be62e53bd221f2ecaeb80f5c3687dc3e04535db"} Jul 10 03:01:02.127350 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:02.127194 2575 scope.go:117] "RemoveContainer" containerID="8ffff42580fbb259b0127a2b9172aeb1541d25b547a1d948cdca1ccce934d981" Jul 10 03:01:02.128768 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:02.128744 2575 generic.go:356] "Generic (PLEG): container finished" podID="99390220-a0ab-45fd-b548-47befe758cb1" containerID="8036608f3d38ab2e87ea24693add913f9e9a973d440ff08c50f548620d91cf7e" exitCode=0 Jul 10 03:01:02.128861 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:02.128796 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7371-139815048301424" event={"ID":"99390220-a0ab-45fd-b548-47befe758cb1","Type":"ContainerDied","Data":"8036608f3d38ab2e87ea24693add913f9e9a973d440ff08c50f548620d91cf7e"} Jul 10 03:01:02.139434 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:02.139288 2575 scope.go:117] "RemoveContainer" containerID="8ffff42580fbb259b0127a2b9172aeb1541d25b547a1d948cdca1ccce934d981" Jul 10 03:01:02.139603 ip-10-0-128-69 kubenswrapper[2575]: E0710 03:01:02.139584 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffff42580fbb259b0127a2b9172aeb1541d25b547a1d948cdca1ccce934d981\": container with ID starting with 8ffff42580fbb259b0127a2b9172aeb1541d25b547a1d948cdca1ccce934d981 not found: ID does not exist" containerID="8ffff42580fbb259b0127a2b9172aeb1541d25b547a1d948cdca1ccce934d981" Jul 10 03:01:02.139644 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:02.139612 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffff42580fbb259b0127a2b9172aeb1541d25b547a1d948cdca1ccce934d981"} err="failed to get container status \"8ffff42580fbb259b0127a2b9172aeb1541d25b547a1d948cdca1ccce934d981\": rpc error: code = NotFound desc = could not find container \"8ffff42580fbb259b0127a2b9172aeb1541d25b547a1d948cdca1ccce934d981\": container with ID starting with 8ffff42580fbb259b0127a2b9172aeb1541d25b547a1d948cdca1ccce934d981 not found: ID does not exist" Jul 10 03:01:02.156870 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:02.156844 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj"] Jul 10 03:01:02.160613 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:02.160594 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1d77d1/rate-test-model-1d77d1-kserve-5fc549d77f-tpplj"] Jul 10 03:01:03.129983 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.129948 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70b48950-184a-46b5-8d9e-0044520dd928" path="/var/lib/kubelet/pods/70b48950-184a-46b5-8d9e-0044520dd928/volumes" Jul 10 03:01:03.277578 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.277550 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048301424" Jul 10 03:01:03.296910 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.296849 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7371-139815048301424_99390220-a0ab-45fd-b548-47befe758cb1/test-curl-7371-139815048301424/0.log" Jul 10 03:01:03.320004 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.319974 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7371-139815048301424"] Jul 10 03:01:03.323359 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.323338 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7371-139815048301424"] Jul 10 03:01:03.331399 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.331379 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdls6\" (UniqueName: \"kubernetes.io/projected/99390220-a0ab-45fd-b548-47befe758cb1-kube-api-access-rdls6\") pod \"99390220-a0ab-45fd-b548-47befe758cb1\" (UID: \"99390220-a0ab-45fd-b548-47befe758cb1\") " Jul 10 03:01:03.333378 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.333356 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99390220-a0ab-45fd-b548-47befe758cb1-kube-api-access-rdls6" (OuterVolumeSpecName: "kube-api-access-rdls6") pod "99390220-a0ab-45fd-b548-47befe758cb1" (UID: "99390220-a0ab-45fd-b548-47befe758cb1"). InnerVolumeSpecName "kube-api-access-rdls6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:01:03.432782 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.432689 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdls6\" (UniqueName: \"kubernetes.io/projected/99390220-a0ab-45fd-b548-47befe758cb1-kube-api-access-rdls6\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 03:01:03.435347 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.435320 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7371-139815048301424"] Jul 10 03:01:03.435726 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.435710 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99390220-a0ab-45fd-b548-47befe758cb1" containerName="test-curl-7371-139815048301424" Jul 10 03:01:03.435796 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.435730 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="99390220-a0ab-45fd-b548-47befe758cb1" containerName="test-curl-7371-139815048301424" Jul 10 03:01:03.435796 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.435753 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70b48950-184a-46b5-8d9e-0044520dd928" containerName="main" Jul 10 03:01:03.435796 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.435762 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="70b48950-184a-46b5-8d9e-0044520dd928" containerName="main" Jul 10 03:01:03.435949 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.435872 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="99390220-a0ab-45fd-b548-47befe758cb1" containerName="test-curl-7371-139815048301424" Jul 10 03:01:03.435949 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.435889 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="70b48950-184a-46b5-8d9e-0044520dd928" containerName="main" Jul 10 03:01:03.438539 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.438522 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048301424" Jul 10 03:01:03.444507 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.444485 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7371-139815048301424"] Jul 10 03:01:03.533347 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.533314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmhkg\" (UniqueName: \"kubernetes.io/projected/73d30d17-8d6a-4954-8d58-b6f9e84a4ca0-kube-api-access-hmhkg\") pod \"test-curl-7371-139815048301424\" (UID: \"73d30d17-8d6a-4954-8d58-b6f9e84a4ca0\") " pod="openshift-ingress/test-curl-7371-139815048301424" Jul 10 03:01:03.634100 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.634064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhkg\" (UniqueName: \"kubernetes.io/projected/73d30d17-8d6a-4954-8d58-b6f9e84a4ca0-kube-api-access-hmhkg\") pod \"test-curl-7371-139815048301424\" (UID: \"73d30d17-8d6a-4954-8d58-b6f9e84a4ca0\") " pod="openshift-ingress/test-curl-7371-139815048301424" Jul 10 03:01:03.642466 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.642436 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmhkg\" (UniqueName: \"kubernetes.io/projected/73d30d17-8d6a-4954-8d58-b6f9e84a4ca0-kube-api-access-hmhkg\") pod \"test-curl-7371-139815048301424\" (UID: \"73d30d17-8d6a-4954-8d58-b6f9e84a4ca0\") " pod="openshift-ingress/test-curl-7371-139815048301424" Jul 10 03:01:03.748127 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.748036 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048301424" Jul 10 03:01:03.879645 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:03.879621 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7371-139815048301424"] Jul 10 03:01:03.881314 ip-10-0-128-69 kubenswrapper[2575]: W0710 03:01:03.881285 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73d30d17_8d6a_4954_8d58_b6f9e84a4ca0.slice/crio-2ee43681a1e86403bcdcc371eab59373bf31256c25c2db1d6ad5d654c89d9989 WatchSource:0}: Error finding container 2ee43681a1e86403bcdcc371eab59373bf31256c25c2db1d6ad5d654c89d9989: Status 404 returned error can't find the container with id 2ee43681a1e86403bcdcc371eab59373bf31256c25c2db1d6ad5d654c89d9989 Jul 10 03:01:04.141642 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:04.141610 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048301424" Jul 10 03:01:04.141642 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:04.141629 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8291e6ba7af78eb1e256dbd35db2cd632714f582c102c5407bc7afb665aea654" Jul 10 03:01:04.142695 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:04.142667 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7371-139815048301424" event={"ID":"73d30d17-8d6a-4954-8d58-b6f9e84a4ca0","Type":"ContainerStarted","Data":"2ee43681a1e86403bcdcc371eab59373bf31256c25c2db1d6ad5d654c89d9989"} Jul 10 03:01:04.152787 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:04.152762 2575 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7371-139815048301424" oldPodUID="99390220-a0ab-45fd-b548-47befe758cb1" podUID="73d30d17-8d6a-4954-8d58-b6f9e84a4ca0" Jul 10 03:01:05.129912 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:05.129867 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99390220-a0ab-45fd-b548-47befe758cb1" path="/var/lib/kubelet/pods/99390220-a0ab-45fd-b548-47befe758cb1/volumes" Jul 10 03:01:05.150763 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:05.150732 2575 generic.go:356] "Generic (PLEG): container finished" podID="73d30d17-8d6a-4954-8d58-b6f9e84a4ca0" containerID="67beb5a2f510222625d054d3b65ab3f1dd92fdea47967ea73761888a1fca9417" exitCode=0 Jul 10 03:01:05.151149 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:05.150824 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7371-139815048301424" event={"ID":"73d30d17-8d6a-4954-8d58-b6f9e84a4ca0","Type":"ContainerDied","Data":"67beb5a2f510222625d054d3b65ab3f1dd92fdea47967ea73761888a1fca9417"} Jul 10 03:01:06.300292 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:06.300268 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048301424" Jul 10 03:01:06.322211 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:06.322186 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7371-139815048301424_73d30d17-8d6a-4954-8d58-b6f9e84a4ca0/test-curl-7371-139815048301424/0.log" Jul 10 03:01:06.346232 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:06.346200 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7371-139815048301424"] Jul 10 03:01:06.349846 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:06.349823 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7371-139815048301424"] Jul 10 03:01:06.359415 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:06.359395 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmhkg\" (UniqueName: \"kubernetes.io/projected/73d30d17-8d6a-4954-8d58-b6f9e84a4ca0-kube-api-access-hmhkg\") pod \"73d30d17-8d6a-4954-8d58-b6f9e84a4ca0\" (UID: \"73d30d17-8d6a-4954-8d58-b6f9e84a4ca0\") " Jul 10 03:01:06.361367 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:06.361348 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d30d17-8d6a-4954-8d58-b6f9e84a4ca0-kube-api-access-hmhkg" (OuterVolumeSpecName: "kube-api-access-hmhkg") pod "73d30d17-8d6a-4954-8d58-b6f9e84a4ca0" (UID: "73d30d17-8d6a-4954-8d58-b6f9e84a4ca0"). InnerVolumeSpecName "kube-api-access-hmhkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:01:06.460089 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:06.460016 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmhkg\" (UniqueName: \"kubernetes.io/projected/73d30d17-8d6a-4954-8d58-b6f9e84a4ca0-kube-api-access-hmhkg\") on node \"ip-10-0-128-69.ec2.internal\" DevicePath \"\"" Jul 10 03:01:07.131076 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:07.131042 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73d30d17-8d6a-4954-8d58-b6f9e84a4ca0" path="/var/lib/kubelet/pods/73d30d17-8d6a-4954-8d58-b6f9e84a4ca0/volumes" Jul 10 03:01:07.162765 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:07.162740 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7371-139815048301424" Jul 10 03:01:07.162898 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:07.162786 2575 scope.go:117] "RemoveContainer" containerID="67beb5a2f510222625d054d3b65ab3f1dd92fdea47967ea73761888a1fca9417" Jul 10 03:01:09.938175 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:09.938131 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 03:01:19.221777 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:19.221744 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-mcpr5"] Jul 10 03:01:25.706785 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:25.706689 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kjvkw_b8838bec-0054-46b7-9fc0-9bf5089f0949/manager/0.log" Jul 10 03:01:25.823230 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:25.823190 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-ps52z_77b501f6-3b1b-4da6-8e58-d3c5c99edcbf/manager/0.log" Jul 10 03:01:26.061960 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:26.061820 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-kxtp7_bb190fb4-b0f8-409b-9c9c-194dc761339b/server/0.log" Jul 10 03:01:26.177820 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:26.177792 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vbqbf_7cae723c-e55b-434c-b2af-b69a0566a41f/manager/0.log" Jul 10 03:01:26.528686 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:26.528654 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc46458b-tkz2d_d7c894dc-5d38-4a54-8448-5bed2c99389b/copy-manifests/0.log" Jul 10 03:01:26.540366 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:26.540340 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc46458b-tkz2d_d7c894dc-5d38-4a54-8448-5bed2c99389b/manager/0.log" Jul 10 03:01:28.166063 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:28.166036 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-558c6fb9c4-zhdr7_6519b348-ea0d-4b30-8773-650b70714231/manager/0.log" Jul 10 03:01:28.274856 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:28.274823 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5546559d7-zvxmj_a7e0c6db-86ef-4053-81dd-0b00748a676d/manager/0.log" Jul 10 03:01:28.383166 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:28.383139 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-24gbv_a3a9346d-6628-43f8-85af-e6e451b7d156/kuadrant-console-plugin/0.log" Jul 10 03:01:28.500866 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:28.500781 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-7569bc7c69-cnt7f_29ef7b85-2eb6-4b3d-8705-b1f76d294e98/manager/0.log" Jul 10 03:01:28.621616 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:28.621585 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-mcpr5_1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2/limitador/0.log" Jul 10 03:01:28.736409 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:28.736381 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-7d6bb9cf99-7jlv4_4f6be0eb-9bac-4347-857a-b88919001b05/manager/0.log" Jul 10 03:01:29.300196 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:29.300170 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f74c66c44-gm26z_301130f0-ce08-428f-9171-f65ca4ca8d98/kube-auth-proxy/0.log" Jul 10 03:01:30.516444 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:30.516416 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7k7gz_a121c9bd-daa5-4a09-b851-8639987315b5/main/0.log" Jul 10 03:01:30.625535 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:30.625506 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ww94k_e93b66a8-cb4e-4b21-8b32-bee9f8daacc4/main/0.log" Jul 10 03:01:30.739428 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:30.739388 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-8ts7h_28cc837f-6d23-449e-9762-7f9ce656d3e6/main/0.log" Jul 10 03:01:37.900018 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:37.899982 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lgw26_82138b66-c754-4c2d-a828-a104a92a9968/global-pull-secret-syncer/0.log" Jul 10 03:01:38.067285 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:38.066672 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mnprz_33cbbbcf-226b-4c91-8f18-f3e5ab0ee24b/konnectivity-agent/0.log" Jul 10 03:01:38.087649 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:38.087610 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-69.ec2.internal_042669854574a08b037a614d9f1b2909/haproxy/0.log" Jul 10 03:01:42.775553 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:42.775525 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-558c6fb9c4-zhdr7_6519b348-ea0d-4b30-8773-650b70714231/manager/0.log" Jul 10 03:01:42.806185 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:42.806152 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5546559d7-zvxmj_a7e0c6db-86ef-4053-81dd-0b00748a676d/manager/0.log" Jul 10 03:01:42.842295 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:42.842269 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-24gbv_a3a9346d-6628-43f8-85af-e6e451b7d156/kuadrant-console-plugin/0.log" Jul 10 03:01:42.984436 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:42.984405 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-7569bc7c69-cnt7f_29ef7b85-2eb6-4b3d-8705-b1f76d294e98/manager/0.log" Jul 10 03:01:43.006754 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:43.006728 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-mcpr5_1c3dfd7c-b199-4dcf-8678-45f81bc3b3b2/limitador/0.log" Jul 10 03:01:43.079841 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:43.079744 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-7d6bb9cf99-7jlv4_4f6be0eb-9bac-4347-857a-b88919001b05/manager/0.log" Jul 10 03:01:44.849126 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:44.849082 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6bcbfc5cb6-flcgc_c9ef7c30-6c22-4d92-b998-1d94ce678b25/metrics-server/0.log" Jul 10 03:01:45.006445 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:45.006409 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lxnqq_66e149b8-86a5-4801-a0df-84b5a98148c8/node-exporter/0.log" Jul 10 03:01:45.028006 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:45.027985 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lxnqq_66e149b8-86a5-4801-a0df-84b5a98148c8/kube-rbac-proxy/0.log" Jul 10 03:01:45.048773 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:45.048752 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lxnqq_66e149b8-86a5-4801-a0df-84b5a98148c8/init-textfile/0.log" Jul 10 03:01:48.580518 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.580483 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck"] Jul 10 03:01:48.580910 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.580872 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73d30d17-8d6a-4954-8d58-b6f9e84a4ca0" containerName="test-curl-7371-139815048301424" Jul 10 03:01:48.580910 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.580890 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="73d30d17-8d6a-4954-8d58-b6f9e84a4ca0" containerName="test-curl-7371-139815048301424" Jul 10 03:01:48.580996 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.580987 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="73d30d17-8d6a-4954-8d58-b6f9e84a4ca0" containerName="test-curl-7371-139815048301424" Jul 10 03:01:48.585531 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.585512 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.588204 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.588182 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2z9m4\"/\"openshift-service-ca.crt\"" Jul 10 03:01:48.588323 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.588208 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2z9m4\"/\"kube-root-ca.crt\"" Jul 10 03:01:48.589216 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.589202 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2z9m4\"/\"default-dockercfg-k6xcg\"" Jul 10 03:01:48.593384 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.593361 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck"] Jul 10 03:01:48.630013 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.629989 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7fdcb8fc5-4lw8r_14d72591-381c-4968-a538-76886c0435f2/console/0.log" Jul 10 03:01:48.632622 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.632600 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2952c16-6a88-47ef-a07b-7320b8b49062-lib-modules\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.632741 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.632648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2952c16-6a88-47ef-a07b-7320b8b49062-proc\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.632741 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.632706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2952c16-6a88-47ef-a07b-7320b8b49062-sys\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.632741 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.632720 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2952c16-6a88-47ef-a07b-7320b8b49062-podres\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.632741 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.632738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdgxb\" (UniqueName: \"kubernetes.io/projected/a2952c16-6a88-47ef-a07b-7320b8b49062-kube-api-access-wdgxb\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.733567 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.733535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdgxb\" (UniqueName: \"kubernetes.io/projected/a2952c16-6a88-47ef-a07b-7320b8b49062-kube-api-access-wdgxb\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.733770 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.733592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2952c16-6a88-47ef-a07b-7320b8b49062-lib-modules\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.733770 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.733660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2952c16-6a88-47ef-a07b-7320b8b49062-proc\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.733770 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.733736 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2952c16-6a88-47ef-a07b-7320b8b49062-sys\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.733770 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.733747 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2952c16-6a88-47ef-a07b-7320b8b49062-lib-modules\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.733958 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.733769 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2952c16-6a88-47ef-a07b-7320b8b49062-proc\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.733958 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.733757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2952c16-6a88-47ef-a07b-7320b8b49062-podres\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.733958 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.733821 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2952c16-6a88-47ef-a07b-7320b8b49062-sys\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.733958 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.733854 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2952c16-6a88-47ef-a07b-7320b8b49062-podres\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.743517 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.743494 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdgxb\" (UniqueName: \"kubernetes.io/projected/a2952c16-6a88-47ef-a07b-7320b8b49062-kube-api-access-wdgxb\") pod \"perf-node-gather-daemonset-dbjck\" (UID: \"a2952c16-6a88-47ef-a07b-7320b8b49062\") " pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:48.896437 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:48.896399 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:49.028593 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:49.028401 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck"] Jul 10 03:01:49.030904 ip-10-0-128-69 kubenswrapper[2575]: W0710 03:01:49.030879 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda2952c16_6a88_47ef_a07b_7320b8b49062.slice/crio-f788cc51b1286853348e6628a15c49a8ae5dd065b46982d6ffd973729b796cfd WatchSource:0}: Error finding container f788cc51b1286853348e6628a15c49a8ae5dd065b46982d6ffd973729b796cfd: Status 404 returned error can't find the container with id f788cc51b1286853348e6628a15c49a8ae5dd065b46982d6ffd973729b796cfd Jul 10 03:01:49.394223 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:49.394178 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" event={"ID":"a2952c16-6a88-47ef-a07b-7320b8b49062","Type":"ContainerStarted","Data":"9c8dfedf4a37c6e2a262961801b94daf2bdaa7257b578ed5112ccc7d11716092"} Jul 10 03:01:49.394748 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:49.394284 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" event={"ID":"a2952c16-6a88-47ef-a07b-7320b8b49062","Type":"ContainerStarted","Data":"f788cc51b1286853348e6628a15c49a8ae5dd065b46982d6ffd973729b796cfd"} Jul 10 03:01:49.394748 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:49.394338 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:49.410064 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:49.410015 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" podStartSLOduration=1.409995892 podStartE2EDuration="1.409995892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 03:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:01:49.408483807 +0000 UTC m=+2338.807936721" watchObservedRunningTime="2026-07-10 03:01:49.409995892 +0000 UTC m=+2338.809448836" Jul 10 03:01:50.453240 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:50.453213 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5drj5_043938f8-88f9-438f-a386-a0bf37ef9b1f/dns/0.log" Jul 10 03:01:50.474816 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:50.474791 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5drj5_043938f8-88f9-438f-a386-a0bf37ef9b1f/kube-rbac-proxy/0.log" Jul 10 03:01:50.617061 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:50.617033 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d47h5_a2929b6d-dfa7-448e-8560-235cddb333e8/dns-node-resolver/0.log" Jul 10 03:01:51.240479 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:51.240449 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hv4z2_62337cd8-e0f1-421f-9b88-e9a4f643841a/node-ca/0.log" Jul 10 03:01:52.448168 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:52.448129 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f74c66c44-gm26z_301130f0-ce08-428f-9171-f65ca4ca8d98/kube-auth-proxy/0.log" Jul 10 03:01:53.306235 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:53.306205 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b79d4_ec1f23d1-33ad-495d-b737-ea19ddc98b69/serve-healthcheck-canary/0.log" Jul 10 03:01:53.791597 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:53.791569 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-fkdsw_0ce04276-ac1b-463f-b812-5dd0b2c82145/insights-operator/0.log" Jul 10 03:01:53.791967 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:53.791846 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-fkdsw_0ce04276-ac1b-463f-b812-5dd0b2c82145/insights-operator/1.log" Jul 10 03:01:53.948259 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:53.948231 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qzjnp_35865d4b-e324-4102-b2c6-0cfee7b6c10d/kube-rbac-proxy/0.log" Jul 10 03:01:53.968921 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:53.968899 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qzjnp_35865d4b-e324-4102-b2c6-0cfee7b6c10d/exporter/0.log" Jul 10 03:01:53.989395 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:53.989370 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qzjnp_35865d4b-e324-4102-b2c6-0cfee7b6c10d/extractor/0.log" Jul 10 03:01:55.410028 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:55.410000 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2z9m4/perf-node-gather-daemonset-dbjck" Jul 10 03:01:56.126210 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:56.126181 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kjvkw_b8838bec-0054-46b7-9fc0-9bf5089f0949/manager/0.log" Jul 10 03:01:56.278975 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:56.278942 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-ps52z_77b501f6-3b1b-4da6-8e58-d3c5c99edcbf/manager/0.log" Jul 10 03:01:56.430219 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:56.430136 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-kxtp7_bb190fb4-b0f8-409b-9c9c-194dc761339b/server/0.log" Jul 10 03:01:56.503408 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:56.503373 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vbqbf_7cae723c-e55b-434c-b2af-b69a0566a41f/manager/0.log" Jul 10 03:01:56.709813 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:56.709729 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc46458b-tkz2d_d7c894dc-5d38-4a54-8448-5bed2c99389b/manager/0.log" Jul 10 03:01:56.730217 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:56.730187 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc46458b-tkz2d_d7c894dc-5d38-4a54-8448-5bed2c99389b/copy-manifests/0.log" Jul 10 03:01:58.107813 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:58.107786 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-24sgd_965ee09b-bb61-4294-865c-32f3429f8c72/openshift-lws-operator/0.log" Jul 10 03:01:59.498835 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:01:59.498805 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f74c66c44-gm26z_301130f0-ce08-428f-9171-f65ca4ca8d98/kube-auth-proxy/0.log" Jul 10 03:02:01.067929 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:01.067893 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f74c66c44-gm26z_301130f0-ce08-428f-9171-f65ca4ca8d98/kube-auth-proxy/0.log" Jul 10 03:02:04.869908 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:04.869882 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sd54c_d74a1be9-254d-4ec7-a66a-e521e4d19dca/kube-multus-additional-cni-plugins/0.log" Jul 10 03:02:04.889826 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:04.889804 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sd54c_d74a1be9-254d-4ec7-a66a-e521e4d19dca/egress-router-binary-copy/0.log" Jul 10 03:02:04.911000 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:04.910979 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sd54c_d74a1be9-254d-4ec7-a66a-e521e4d19dca/cni-plugins/0.log" Jul 10 03:02:04.931882 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:04.931862 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sd54c_d74a1be9-254d-4ec7-a66a-e521e4d19dca/bond-cni-plugin/0.log" Jul 10 03:02:04.953172 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:04.953153 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sd54c_d74a1be9-254d-4ec7-a66a-e521e4d19dca/routeoverride-cni/0.log" Jul 10 03:02:04.974816 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:04.974793 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sd54c_d74a1be9-254d-4ec7-a66a-e521e4d19dca/whereabouts-cni-bincopy/0.log" Jul 10 03:02:04.996903 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:04.996885 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sd54c_d74a1be9-254d-4ec7-a66a-e521e4d19dca/whereabouts-cni/0.log" Jul 10 03:02:05.187450 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:05.187374 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-blpm7_3ed8f1f1-c480-41a9-b2f6-2bfabe74c7ee/kube-multus/0.log" Jul 10 03:02:05.333129 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:05.333103 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r4t9z_90c11f6c-5762-4634-9dad-5dc5e5b751eb/network-metrics-daemon/0.log" Jul 10 03:02:05.351852 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:05.351830 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r4t9z_90c11f6c-5762-4634-9dad-5dc5e5b751eb/kube-rbac-proxy/0.log" Jul 10 03:02:06.442165 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:06.442132 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-controller/0.log" Jul 10 03:02:06.461935 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:06.461897 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/1.log" Jul 10 03:02:06.461935 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:06.461907 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/ovn-acl-logging/0.log" Jul 10 03:02:06.478424 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:06.478389 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/kube-rbac-proxy-node/0.log" Jul 10 03:02:06.500801 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:06.500777 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/kube-rbac-proxy-ovn-metrics/0.log" Jul 10 03:02:06.521864 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:06.521831 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/northd/0.log" Jul 10 03:02:06.544159 ip-10-0-128-69 kubenswrapper[2575]: I0710 03:02:06.544134 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlm7j_61241733-b531-4151-a869-cc89000e2e99/nbdb/0.log"