Jul 08 02:56:19.706986 ip-10-0-142-99 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 08 02:56:19.706998 ip-10-0-142-99 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 08 02:56:19.707005 ip-10-0-142-99 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 08 02:56:19.707245 ip-10-0-142-99 systemd[1]: Failed to start Kubernetes Kubelet. Jul 08 02:56:29.731799 ip-10-0-142-99 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 08 02:56:29.731813 ip-10-0-142-99 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 186d1187757c47e194531677768b67a3 -- Jul 08 02:57:59.824611 ip-10-0-142-99 systemd[1]: Starting Kubernetes Kubelet... Jul 08 02:58:00.242163 ip-10-0-142-99 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 02:58:00.242163 ip-10-0-142-99 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 02:58:00.242163 ip-10-0-142-99 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 02:58:00.242163 ip-10-0-142-99 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 02:58:00.242163 ip-10-0-142-99 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 02:58:00.243443 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.243281 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 02:58:00.248115 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248097 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:58:00.248115 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248115 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:58:00.248115 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248120 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248124 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248128 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248132 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248135 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248139 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248142 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248146 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248150 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248154 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248163 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248168 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248171 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248175 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248180 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248185 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248189 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248193 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248197 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248201 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:58:00.248386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248205 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248209 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248213 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248217 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248221 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248225 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248229 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248233 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248237 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248241 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248246 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248250 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248254 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248258 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248263 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248267 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248273 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248277 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248281 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248286 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:58:00.249193 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248290 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248295 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248319 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248325 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248329 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248333 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248337 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248342 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248350 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248357 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248361 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248366 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248370 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248374 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248378 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248382 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248386 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248392 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248397 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:58:00.250183 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248404 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248409 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248413 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248418 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248422 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248426 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248430 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248436 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248441 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248445 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248449 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248453 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248457 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248462 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248466 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248470 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248476 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248483 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248487 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248491 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:58:00.250986 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248495 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248499 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248503 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248507 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248512 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248516 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248520 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248524 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248528 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248532 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248536 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248541 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248545 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248549 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248553 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248558 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248562 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.248566 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249846 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249859 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:58:00.251857 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249864 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249868 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249874 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249880 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249885 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249889 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249893 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249897 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249901 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249905 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249911 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249915 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249920 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249924 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249928 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249932 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249936 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249941 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249945 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:58:00.252765 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249949 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249954 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249958 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249962 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249966 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249970 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249974 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249978 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249982 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249986 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249991 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.249995 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250000 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250004 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250008 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250012 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250016 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250021 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250026 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250031 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:58:00.253431 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250035 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250039 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250043 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250047 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250059 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250064 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250068 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250073 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250077 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250081 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250085 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250089 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250094 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250098 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250101 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250105 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250109 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250113 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250118 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:58:00.253990 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250122 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250126 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250130 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250134 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250138 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250144 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250149 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250153 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250161 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250166 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250170 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250174 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250178 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250182 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250185 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250189 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250193 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250197 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250214 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:58:00.254554 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250220 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250226 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250231 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250235 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250239 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250243 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250247 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250251 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250255 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250259 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250264 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250268 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250272 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250276 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250281 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250285 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250290 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250294 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250298 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.250320 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:58:00.255252 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251112 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251122 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251126 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251131 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251136 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251140 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251145 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251150 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251154 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251158 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251162 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251167 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251172 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251176 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251179 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251183 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251187 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251192 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251195 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:58:00.255760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251200 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251204 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251208 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251213 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251217 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251222 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251225 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251229 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251233 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251238 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251242 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251246 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251250 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251254 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251258 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251262 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251267 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251271 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251275 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251279 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:58:00.256235 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251283 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251287 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251291 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251295 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251318 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251324 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251328 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251332 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251336 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251340 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251343 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251347 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251352 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251356 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251360 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251364 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251369 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251373 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251376 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:58:00.256825 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251380 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251387 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251394 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251399 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251403 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251407 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251412 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251416 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251421 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251426 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251430 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251435 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251440 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251444 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251448 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251452 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251457 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251461 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251465 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251470 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:58:00.257284 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251478 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251483 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251488 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251492 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251495 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251499 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251503 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251507 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251512 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251516 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251523 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251528 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251533 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251538 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251542 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251547 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251551 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251556 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251560 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:58:00.257819 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251564 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.251568 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251661 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251671 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251685 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251702 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251709 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251715 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251722 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251732 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251737 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251742 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251748 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251754 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251760 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251765 2576 flags.go:64] FLAG: --cgroup-root="" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251769 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251774 2576 flags.go:64] FLAG: --client-ca-file="" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251778 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251783 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251791 2576 flags.go:64] FLAG: --cluster-domain="" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251796 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251801 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251805 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 02:58:00.258270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251811 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251817 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251821 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251826 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251832 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251836 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251841 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251846 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251851 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251856 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251862 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251867 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251871 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251875 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251879 2576 flags.go:64] FLAG: --enable-server="true" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251884 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251891 2576 flags.go:64] FLAG: --event-burst="100" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251895 2576 flags.go:64] FLAG: --event-qps="50" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251900 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251905 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251909 2576 flags.go:64] FLAG: --eviction-hard="" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251915 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251921 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251925 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251930 2576 flags.go:64] FLAG: --eviction-soft="" Jul 08 02:58:00.258854 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251934 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251939 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251943 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251948 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251955 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251960 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251966 2576 flags.go:64] FLAG: --feature-gates="" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251972 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251977 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251982 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251990 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.251995 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252000 2576 flags.go:64] FLAG: --help="false" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252005 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252010 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252014 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252019 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252024 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252030 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252035 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252039 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252044 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252048 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 02:58:00.259464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252053 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252058 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252062 2576 flags.go:64] FLAG: --kube-reserved="" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252067 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252071 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252077 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252081 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252087 2576 flags.go:64] FLAG: --lock-file="" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252091 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252096 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252101 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252110 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252114 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252119 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252124 2576 flags.go:64] FLAG: --logging-format="text" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252129 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252134 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252138 2576 flags.go:64] FLAG: --manifest-url="" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252143 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252152 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252157 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252164 2576 flags.go:64] FLAG: --max-pods="110" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252168 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252173 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252178 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 02:58:00.260099 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252182 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252187 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252192 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252196 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252207 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252212 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252217 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252221 2576 flags.go:64] FLAG: --pod-cidr="" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252226 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252234 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252239 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252244 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252249 2576 flags.go:64] FLAG: --port="10250" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252254 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252258 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-037d443cf68f4cc87" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252264 2576 flags.go:64] FLAG: --qos-reserved="" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252269 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252274 2576 flags.go:64] FLAG: --register-node="true" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252279 2576 flags.go:64] FLAG: --register-with-taints="" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252285 2576 flags.go:64] FLAG: --registry-burst="10" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252289 2576 flags.go:64] FLAG: --registry-qps="5" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252294 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252317 2576 flags.go:64] FLAG: --reserved-memory="" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252324 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252329 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 02:58:00.260799 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252334 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252338 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252345 2576 flags.go:64] FLAG: --runonce="false" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252349 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252355 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252360 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252365 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252369 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252374 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252379 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252384 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252389 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252394 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252399 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252403 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252408 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252412 2576 flags.go:64] FLAG: --system-cgroups="" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252417 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252426 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252432 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252437 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252444 2576 flags.go:64] FLAG: --tls-min-version="" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252449 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252454 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252459 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 02:58:00.261670 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252464 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252469 2576 flags.go:64] FLAG: --v="2" Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252475 2576 flags.go:64] FLAG: --version="false" Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252482 2576 flags.go:64] FLAG: --vmodule="" Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252488 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.252493 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252690 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252697 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252704 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252712 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252719 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252723 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252727 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252732 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252736 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252740 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252744 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252748 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252752 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252756 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252761 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:58:00.262406 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252764 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252768 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252773 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252776 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252780 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252784 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252788 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252792 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252796 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252800 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252805 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252809 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252813 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252818 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252822 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252826 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252832 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252837 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252842 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252846 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:58:00.262915 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252853 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252859 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252864 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252868 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252873 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252878 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252882 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252888 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252892 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252897 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252901 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252906 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252910 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252913 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252917 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252921 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252925 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252930 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252933 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252938 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:58:00.263417 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252942 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252946 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252950 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252955 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252960 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252964 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252968 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252972 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252976 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252980 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252984 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252989 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.252994 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253000 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253005 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253009 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253013 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253017 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253021 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253025 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:58:00.263888 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253030 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253034 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253038 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253042 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253047 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253051 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253055 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253059 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253063 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253067 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253071 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253075 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253079 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253085 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253090 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253094 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253099 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253104 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253108 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:58:00.264428 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253112 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:58:00.264889 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253116 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:58:00.264889 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253120 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:58:00.264889 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253124 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:58:00.264889 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.253128 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:58:00.264889 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.253859 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 02:58:00.264889 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.263714 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 02:58:00.264889 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.263731 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 02:58:00.264889 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.264777 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 02:58:00.264889 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.264786 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 02:58:00.264889 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264852 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:58:00.264889 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264857 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:58:00.264889 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264860 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:58:00.264889 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264863 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:58:00.264889 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264866 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264869 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264871 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264874 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264877 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264880 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264882 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264885 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264887 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264890 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264892 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264895 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264898 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264901 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264904 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264907 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264910 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264912 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264915 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264917 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:58:00.265240 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264919 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264922 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264925 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264927 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264930 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264933 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264936 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264938 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264940 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264943 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264946 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264948 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264951 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264953 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264957 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264960 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264962 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264965 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264968 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264970 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:58:00.265746 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264972 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264976 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264980 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264983 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264986 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264989 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264991 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264994 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.264997 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265001 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265003 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265006 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265009 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265013 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265015 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265018 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265020 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265022 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265025 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265028 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:58:00.266227 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265030 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265033 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265036 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265038 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265041 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265043 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265046 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265049 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265051 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265054 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265056 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265059 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265061 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265064 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265067 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265070 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265072 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265075 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265077 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:58:00.266741 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265079 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265083 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265087 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265090 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265093 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265095 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265098 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265101 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265103 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265105 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265108 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265110 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265113 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265115 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265119 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:58:00.267198 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265123 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.265128 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265249 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265256 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265261 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265264 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265267 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265270 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265272 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265275 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265278 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265281 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265284 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265286 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265291 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:58:00.267614 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265295 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265298 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265317 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265320 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265322 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265325 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265328 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265331 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265335 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265338 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265341 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265343 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265346 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265348 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265351 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265354 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265356 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265358 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265362 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:58:00.267983 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265365 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265368 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265372 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265375 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265377 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265380 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265383 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265386 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265388 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265391 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265393 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265395 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265398 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265402 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265404 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265407 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265409 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265412 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265414 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:58:00.268462 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265416 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265419 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265421 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265424 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265426 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265429 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265431 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265434 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265436 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265438 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265441 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265443 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265446 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265448 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265451 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265454 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265456 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265458 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265461 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265463 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:58:00.268932 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265466 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265468 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265470 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265473 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265475 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265477 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265480 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265483 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265485 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265488 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265490 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265492 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265495 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265497 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265499 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265502 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265504 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265507 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265509 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265512 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:58:00.269465 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265514 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:58:00.269943 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265516 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:58:00.269943 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265519 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:58:00.269943 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265521 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:58:00.269943 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265524 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:58:00.269943 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265526 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:58:00.269943 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265529 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:58:00.269943 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:00.265531 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:58:00.269943 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.265536 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 02:58:00.269943 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.265677 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 02:58:00.269943 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.269510 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 02:58:00.270989 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.270976 2576 server.go:1019] "Starting client certificate rotation" Jul 08 02:58:00.271097 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.271081 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 02:58:00.271857 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.271845 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 02:58:00.273606 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.273593 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 02:58:00.275024 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.275006 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 02:58:00.287213 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.287197 2576 log.go:25] "Validated CRI v1 runtime API" Jul 08 02:58:00.291454 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.291409 2576 log.go:25] "Validated CRI v1 image API" Jul 08 02:58:00.293205 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.293189 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 02:58:00.295836 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.295814 2576 fs.go:135] Filesystem UUIDs: map[393b435a-a0a5-442c-b718-6720eff0b41f:/dev/nvme0n1p3 6eb82001-a807-4843-9ceb-5b974a52fbe4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 08 02:58:00.295908 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.295835 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 02:58:00.302576 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.302558 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 02:58:00.303431 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.303319 2576 manager.go:216] Machine: {Timestamp:2026-07-08 02:58:00.301921461 +0000 UTC m=+0.369110109 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096081 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c5fbc238871e3ce58c4be86927453 SystemUUID:ec2c5fbc-2388-71e3-ce58-c4be86927453 BootID:186d1187-757c-47e1-9453-1677768b67a3 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:42:22:73:d1:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:42:22:73:d1:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:34:6b:a9:b1:93 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 02:58:00.303431 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.303431 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 02:58:00.303536 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.303525 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 02:58:00.306772 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.306744 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 02:58:00.306918 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.306776 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 02:58:00.307874 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.307862 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 02:58:00.307911 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.307876 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 02:58:00.307911 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.307884 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 02:58:00.308837 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.308826 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 02:58:00.310376 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.310366 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 02:58:00.313969 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.313957 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 08 02:58:00.314101 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.314090 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 02:58:00.314134 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.314124 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 08 02:58:00.314164 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.314137 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 02:58:00.314191 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.314165 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 02:58:00.314191 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.314180 2576 kubelet.go:398] "Adding apiserver pod source" Jul 08 02:58:00.314248 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.314194 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 02:58:00.314648 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.314637 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 02:58:00.314685 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.314656 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 02:58:00.316872 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.316853 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 02:58:00.319494 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.319477 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 02:58:00.319581 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.319515 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 02:58:00.321144 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.321129 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 02:58:00.321219 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.321149 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 02:58:00.321219 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.321158 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 02:58:00.321219 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.321165 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 02:58:00.321219 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.321178 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 02:58:00.321219 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.321188 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 02:58:00.321219 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.321201 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 02:58:00.321219 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.321210 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 02:58:00.321219 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.321220 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 02:58:00.321477 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.321229 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 02:58:00.321477 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.321250 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 02:58:00.321477 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.321260 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 02:58:00.322085 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.322057 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 02:58:00.322085 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.322071 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 02:58:00.325345 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.325330 2576 server.go:1268] "Started kubelet" Jul 08 02:58:00.325991 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.325953 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 02:58:00.326064 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.326043 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 02:58:00.326116 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.326059 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 02:58:00.326169 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.326129 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 02:58:00.326316 ip-10-0-142-99 systemd[1]: Started Kubernetes Kubelet. Jul 08 02:58:00.326826 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.325485 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 02:58:00.326826 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.326554 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 02:58:00.328362 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.328343 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 02:58:00.328965 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.328953 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 08 02:58:00.331059 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.331039 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 02:58:00.331968 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.331009 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-99.ec2.internal.18c0312358fcae2b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-99.ec2.internal,UID:ip-10-0-142-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-99.ec2.internal,},FirstTimestamp:2026-07-08 02:58:00.325287467 +0000 UTC m=+0.392476114,LastTimestamp:2026-07-08 02:58:00.325287467 +0000 UTC m=+0.392476114,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-99.ec2.internal,}" Jul 08 02:58:00.336373 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.336357 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 02:58:00.336445 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.336406 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 02:58:00.336524 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.336484 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found" Jul 08 02:58:00.337121 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.337102 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 02:58:00.337486 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.337466 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 02:58:00.337598 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.337544 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 02:58:00.337794 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.337778 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 08 02:58:00.337794 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.337791 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 08 02:58:00.338778 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.338752 2576 factory.go:55] Registering systemd factory Jul 08 02:58:00.338778 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.338772 2576 factory.go:223] Registration of the systemd container factory successfully Jul 08 02:58:00.338995 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.338980 2576 factory.go:153] Registering CRI-O factory Jul 08 02:58:00.339050 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.338999 2576 factory.go:223] Registration of the crio container factory successfully Jul 08 02:58:00.339050 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.339044 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 02:58:00.339147 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.339068 2576 factory.go:103] Registering Raw factory Jul 08 02:58:00.339147 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.339083 2576 manager.go:1196] Started watching for new ooms in manager Jul 08 02:58:00.339749 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.339718 2576 manager.go:349] Starting recovery of all containers Jul 08 02:58:00.341556 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.341526 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 02:58:00.341689 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.341665 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 02:58:00.350035 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.349862 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rkr7p" Jul 08 02:58:00.350659 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.350644 2576 manager.go:354] Recovery completed Jul 08 02:58:00.355054 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.355040 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:58:00.356891 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.356875 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:58:00.356964 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.356908 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:58:00.356964 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.356923 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:58:00.357430 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.357416 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 02:58:00.357430 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.357430 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 02:58:00.357530 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.357445 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 08 02:58:00.357530 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.357473 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:58:00.357685 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.357669 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rkr7p" Jul 08 02:58:00.358246 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.358232 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:58:00.358335 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.358258 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:58:00.358335 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.358269 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:58:00.358741 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.358731 2576 policy_none.go:49] "None policy: Start" Jul 08 02:58:00.358774 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.358746 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 02:58:00.358774 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.358759 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 02:58:00.359201 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.359127 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-99.ec2.internal.18c031235adee9dc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-99.ec2.internal,UID:ip-10-0-142-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-99.ec2.internal,},FirstTimestamp:2026-07-08 02:58:00.3568911 +0000 UTC m=+0.424079747,LastTimestamp:2026-07-08 02:58:00.3568911 +0000 UTC m=+0.424079747,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-99.ec2.internal,}" Jul 08 02:58:00.359697 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.359685 2576 policy_none.go:47] "Start" Jul 08 02:58:00.401949 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.401930 2576 manager.go:337] "Starting Device Plugin manager" Jul 08 02:58:00.412557 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.401972 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 02:58:00.412557 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.401989 2576 server.go:87] "Starting device plugin registration server" Jul 08 02:58:00.412557 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.402261 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 02:58:00.412557 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.402272 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 02:58:00.412557 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.402367 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 02:58:00.412557 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.402452 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 02:58:00.412557 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.402461 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 02:58:00.412557 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.403115 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 02:58:00.412557 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.403157 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-99.ec2.internal\" not found" Jul 08 02:58:00.440946 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.440918 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 02:58:00.442342 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.442326 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 02:58:00.442414 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.442362 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 02:58:00.442414 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.442387 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 02:58:00.442485 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.442428 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 02:58:00.444537 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.444521 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:58:00.503446 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.503368 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:58:00.504798 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.504778 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:58:00.504904 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.504820 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:58:00.504904 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.504837 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:58:00.504904 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.504868 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.519760 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.519729 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.519760 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.519757 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-99.ec2.internal\": node \"ip-10-0-142-99.ec2.internal\" not found" Jul 08 02:58:00.542668 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.542625 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal"] Jul 08 02:58:00.542808 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.542720 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:58:00.543267 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.543247 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found" Jul 08 02:58:00.544259 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.544241 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:58:00.544382 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.544277 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:58:00.544382 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.544292 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:58:00.545786 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.545769 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:58:00.545881 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.545867 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.545881 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.545892 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:58:00.546585 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.546568 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:58:00.546661 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.546595 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:58:00.546661 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.546605 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:58:00.546661 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.546569 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:58:00.546781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.546668 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:58:00.546781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.546678 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:58:00.548091 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.548075 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.548142 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.548108 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:58:00.548843 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.548828 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:58:00.548905 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.548855 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:58:00.548905 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.548868 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:58:00.576052 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.576029 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-99.ec2.internal\" not found" node="ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.580579 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.580560 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-99.ec2.internal\" not found" node="ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.640079 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.640044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a205a6cae71405e2b5080f71ebefef8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal\" (UID: \"1a205a6cae71405e2b5080f71ebefef8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.640079 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.640080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a205a6cae71405e2b5080f71ebefef8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal\" (UID: \"1a205a6cae71405e2b5080f71ebefef8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.640277 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.640105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b8d67d3d75c39b81a69b1a068c27f3b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-99.ec2.internal\" (UID: \"8b8d67d3d75c39b81a69b1a068c27f3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.643916 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.643893 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found" Jul 08 02:58:00.740890 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.740800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a205a6cae71405e2b5080f71ebefef8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal\" (UID: \"1a205a6cae71405e2b5080f71ebefef8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.741019 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.740919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a205a6cae71405e2b5080f71ebefef8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal\" (UID: \"1a205a6cae71405e2b5080f71ebefef8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.741019 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.740944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b8d67d3d75c39b81a69b1a068c27f3b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-99.ec2.internal\" (UID: \"8b8d67d3d75c39b81a69b1a068c27f3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.741019 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.740859 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a205a6cae71405e2b5080f71ebefef8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal\" (UID: \"1a205a6cae71405e2b5080f71ebefef8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.741019 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.740996 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a205a6cae71405e2b5080f71ebefef8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal\" (UID: \"1a205a6cae71405e2b5080f71ebefef8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.741019 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.741005 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b8d67d3d75c39b81a69b1a068c27f3b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-99.ec2.internal\" (UID: \"8b8d67d3d75c39b81a69b1a068c27f3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.744921 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.744902 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found" Jul 08 02:58:00.845718 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.845686 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found" Jul 08 02:58:00.879888 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.879848 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.884160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:00.884029 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal" Jul 08 02:58:00.946258 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:00.946225 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found" Jul 08 02:58:01.046717 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.046697 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found" Jul 08 02:58:01.147353 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.147262 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found" Jul 08 02:58:01.247835 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.247802 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found" Jul 08 02:58:01.269906 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.269882 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 02:58:01.270082 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.270053 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 02:58:01.309316 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.309283 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:58:01.314519 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.314506 2576 apiserver.go:52] "Watching apiserver" Jul 08 02:58:01.324725 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.324705 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 02:58:01.325063 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.325041 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-h69hf","openshift-multus/multus-additional-cni-plugins-nf95s","openshift-multus/network-metrics-daemon-wdvzz","openshift-network-diagnostics/network-check-target-bwz4j","openshift-network-operator/iptables-alerter-lmfzb","kube-system/konnectivity-agent-9sj9r","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92","openshift-cluster-node-tuning-operator/tuned-27wpf","openshift-image-registry/node-ca-57xjh","openshift-multus/multus-q2md7","openshift-ovn-kubernetes/ovnkube-node-7qb87","kube-system/global-pull-secret-syncer-78f5d"] Jul 08 02:58:01.326553 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.326538 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h69hf" Jul 08 02:58:01.327588 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.327570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.328700 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.328682 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:01.328786 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.328766 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:01.329038 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.329024 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lxsm9\"" Jul 08 02:58:01.329208 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.329187 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 02:58:01.329318 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.329269 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 02:58:01.329779 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.329763 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lmfzb" Jul 08 02:58:01.330216 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.330197 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 02:58:01.330327 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.330240 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 02:58:01.330869 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.330855 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9sj9r" Jul 08 02:58:01.331064 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.331044 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 02:58:01.331984 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.331965 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.332103 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.332089 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 02:58:01.332204 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.332187 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9d7lj\"" Jul 08 02:58:01.332266 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.332224 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 02:58:01.332266 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.332231 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h2vcg\"" Jul 08 02:58:01.332377 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.332321 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 02:58:01.332435 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.332379 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 02:58:01.332435 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.332382 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 02:58:01.333225 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.333211 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 02:58:01.333225 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.333229 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.333364 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.333332 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 02:58:01.333464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.333451 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fms2t\"" Jul 08 02:58:01.334294 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.334279 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 02:58:01.334430 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.334416 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 02:58:01.334582 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.334570 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v22j4\"" Jul 08 02:58:01.334678 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.334663 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.335889 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.335874 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 02:58:01.335986 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.335967 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6rhz5\"" Jul 08 02:58:01.336260 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.336248 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:01.336343 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.336326 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:01.336774 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.336762 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal" Jul 08 02:58:01.336970 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.336949 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 02:58:01.337054 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.336984 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 02:58:01.337101 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.337066 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 02:58:01.337141 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.337116 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 02:58:01.338432 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.338252 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 02:58:01.338842 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.338781 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5bjbp\"" Jul 08 02:58:01.339396 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.339349 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 02:58:01.340493 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.340408 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.340493 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.340412 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 02:58:01.342107 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.341831 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-57xjh" Jul 08 02:58:01.343034 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343017 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 02:58:01.343388 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343371 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 02:58:01.343485 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:01.343485 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343375 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-84b7w\"" Jul 08 02:58:01.343485 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.343474 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:01.343743 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343731 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 02:58:01.343777 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-ovnkube-script-lib\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.343777 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98dr2\" (UniqueName: \"kubernetes.io/projected/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-kube-api-access-98dr2\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.343850 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97edc872-75d9-449b-9fa3-0871dc3a965e-cni-binary-copy\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.343850 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-run-k8s-cni-cncf-io\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.343850 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343833 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-run-netns\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.343850 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4jt2\" (UniqueName: \"kubernetes.io/projected/f4c0c541-bfc7-4376-a46f-622737060480-kube-api-access-b4jt2\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.343962 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-cnibin\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.343962 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343907 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-multus-conf-dir\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.343962 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-slash\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.343962 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.343962 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e25aa2e-b99d-492f-abb2-46c3c8fc0662-agent-certs\") pod \"konnectivity-agent-9sj9r\" (UID: \"4e25aa2e-b99d-492f-abb2-46c3c8fc0662\") " pod="kube-system/konnectivity-agent-9sj9r" Jul 08 02:58:01.344154 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.343976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e25aa2e-b99d-492f-abb2-46c3c8fc0662-konnectivity-ca\") pod \"konnectivity-agent-9sj9r\" (UID: \"4e25aa2e-b99d-492f-abb2-46c3c8fc0662\") " pod="kube-system/konnectivity-agent-9sj9r" Jul 08 02:58:01.344154 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-modprobe-d\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.344154 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344017 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-os-release\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.344154 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344031 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-run-netns\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.344154 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-run-openvswitch\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.344154 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344065 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-run-multus-certs\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.344154 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-etc-openvswitch\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.344154 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344144 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-sys\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.344523 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53a5c28-a70a-4388-bb82-03cd3832d27a-tmp-dir\") pod \"node-resolver-h69hf\" (UID: \"b53a5c28-a70a-4388-bb82-03cd3832d27a\") " pod="openshift-dns/node-resolver-h69hf" Jul 08 02:58:01.344523 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344193 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2r67\" (UniqueName: \"kubernetes.io/projected/b53a5c28-a70a-4388-bb82-03cd3832d27a-kube-api-access-q2r67\") pod \"node-resolver-h69hf\" (UID: \"b53a5c28-a70a-4388-bb82-03cd3832d27a\") " pod="openshift-dns/node-resolver-h69hf" Jul 08 02:58:01.344523 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344222 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-kubelet\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.344523 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-log-socket\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.344523 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.344523 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-sys-fs\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.344523 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab087428-a2bf-4ca5-8c12-1824860547c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.344523 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344372 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-lib-modules\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.344523 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-systemd-units\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.344523 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8g7t\" (UniqueName: \"kubernetes.io/projected/ab087428-a2bf-4ca5-8c12-1824860547c2-kube-api-access-r8g7t\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.344523 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-system-cni-dir\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.344523 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-var-lib-cni-bin\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.344523 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-var-lib-cni-multus\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-var-lib-kubelet\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc7qf\" (UniqueName: \"kubernetes.io/projected/97edc872-75d9-449b-9fa3-0871dc3a965e-kube-api-access-vc7qf\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-var-lib-openvswitch\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-cni-netd\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344634 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab087428-a2bf-4ca5-8c12-1824860547c2-system-cni-dir\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab087428-a2bf-4ca5-8c12-1824860547c2-os-release\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344669 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-multus-cni-dir\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-node-log\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-ovnkube-config\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-env-overrides\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2d44\" (UniqueName: \"kubernetes.io/projected/c392372d-4d45-4636-8058-57fd87930e8e-kube-api-access-s2d44\") pod \"network-metrics-daemon-wdvzz\" (UID: \"c392372d-4d45-4636-8058-57fd87930e8e\") " pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/658616f8-26f1-4236-8be0-908fb5650a90-iptables-alerter-script\") pod \"iptables-alerter-lmfzb\" (UID: \"658616f8-26f1-4236-8be0-908fb5650a90\") " pod="openshift-network-operator/iptables-alerter-lmfzb" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/658616f8-26f1-4236-8be0-908fb5650a90-host-slash\") pod \"iptables-alerter-lmfzb\" (UID: \"658616f8-26f1-4236-8be0-908fb5650a90\") " pod="openshift-network-operator/iptables-alerter-lmfzb" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b53a5c28-a70a-4388-bb82-03cd3832d27a-hosts-file\") pod \"node-resolver-h69hf\" (UID: \"b53a5c28-a70a-4388-bb82-03cd3832d27a\") " pod="openshift-dns/node-resolver-h69hf" Jul 08 02:58:01.344932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-run-ovn\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab087428-a2bf-4ca5-8c12-1824860547c2-cnibin\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4tj2\" (UniqueName: \"kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2\") pod \"network-check-target-bwz4j\" (UID: \"29273a5e-bb56-4d02-9905-33accc2373ef\") " pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-kubernetes\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-etc-selinux\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab087428-a2bf-4ca5-8c12-1824860547c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4j9\" (UniqueName: \"kubernetes.io/projected/658616f8-26f1-4236-8be0-908fb5650a90-kube-api-access-bx4j9\") pod \"iptables-alerter-lmfzb\" (UID: \"658616f8-26f1-4236-8be0-908fb5650a90\") " pod="openshift-network-operator/iptables-alerter-lmfzb" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4c0c541-bfc7-4376-a46f-622737060480-tmp\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.344982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97edc872-75d9-449b-9fa3-0871dc3a965e-multus-daemon-config\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345009 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-sysctl-conf\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab087428-a2bf-4ca5-8c12-1824860547c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-run\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345051 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4c0c541-bfc7-4376-a46f-622737060480-etc-tuned\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-cni-bin\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs\") pod \"network-metrics-daemon-wdvzz\" (UID: \"c392372d-4d45-4636-8058-57fd87930e8e\") " pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-registration-dir\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.345463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345108 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-device-dir\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-sysctl-d\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-etc-kubernetes\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-socket-dir\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqv8w\" (UniqueName: \"kubernetes.io/projected/8d449419-ae00-4fe7-b11f-3c785ea367fc-kube-api-access-qqv8w\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345189 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-run-systemd\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab087428-a2bf-4ca5-8c12-1824860547c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-sysconfig\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345238 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-systemd\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-var-lib-kubelet\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345277 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-multus-socket-dir-parent\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345352 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-hostroot\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-ovn-node-metrics-cert\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345406 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-host\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345437 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345530 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-54l6t\"" Jul 08 02:58:01.345899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.345800 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 02:58:01.346452 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.346439 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 02:58:01.350863 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.350845 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal"] Jul 08 02:58:01.352775 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.352750 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 02:58:01.352848 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.352805 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" Jul 08 02:58:01.357571 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.357555 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 02:58:01.359641 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.359605 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 02:53:01 +0000 UTC" deadline="2028-01-05 02:13:33.167318135 +0000 UTC" Jul 08 02:58:01.359705 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.359642 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13103h15m31.807679613s" Jul 08 02:58:01.372037 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.372024 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 02:58:01.372364 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.372350 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal"] Jul 08 02:58:01.437821 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.437803 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 02:58:01.445532 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-sysconfig\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.445642 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-systemd\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.445642 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445553 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-var-lib-kubelet\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.445642 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-multus-socket-dir-parent\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.445642 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-hostroot\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.445825 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445670 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-sysconfig\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.445825 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445674 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-systemd\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.445825 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-ovn-node-metrics-cert\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.445825 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445724 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-multus-socket-dir-parent\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.445825 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445730 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-var-lib-kubelet\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.445825 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-host\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.445825 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445784 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-hostroot\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.445825 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445794 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-host\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-ovnkube-script-lib\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98dr2\" (UniqueName: \"kubernetes.io/projected/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-kube-api-access-98dr2\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445895 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0343e166-1101-45b0-a57e-cc26b4858985-kubelet-config\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445921 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97edc872-75d9-449b-9fa3-0871dc3a965e-cni-binary-copy\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-run-k8s-cni-cncf-io\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-run-netns\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.445992 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-run-k8s-cni-cncf-io\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4jt2\" (UniqueName: \"kubernetes.io/projected/f4c0c541-bfc7-4376-a46f-622737060480-kube-api-access-b4jt2\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-cnibin\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446079 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-run-netns\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446095 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-cnibin\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-multus-conf-dir\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-slash\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446133 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-multus-conf-dir\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446171 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb95t\" (UniqueName: \"kubernetes.io/projected/97e0ac5b-7e47-49c6-a320-6f45f632c3d4-kube-api-access-sb95t\") pod \"node-ca-57xjh\" (UID: \"97e0ac5b-7e47-49c6-a320-6f45f632c3d4\") " pod="openshift-image-registry/node-ca-57xjh" Jul 08 02:58:01.446180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446172 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-slash\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e25aa2e-b99d-492f-abb2-46c3c8fc0662-agent-certs\") pod \"konnectivity-agent-9sj9r\" (UID: \"4e25aa2e-b99d-492f-abb2-46c3c8fc0662\") " pod="kube-system/konnectivity-agent-9sj9r" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e25aa2e-b99d-492f-abb2-46c3c8fc0662-konnectivity-ca\") pod \"konnectivity-agent-9sj9r\" (UID: \"4e25aa2e-b99d-492f-abb2-46c3c8fc0662\") " pod="kube-system/konnectivity-agent-9sj9r" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446226 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-modprobe-d\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-os-release\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446324 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-run-netns\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-run-openvswitch\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-run-multus-certs\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-etc-openvswitch\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-sys\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446408 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-os-release\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446420 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-modprobe-d\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53a5c28-a70a-4388-bb82-03cd3832d27a-tmp-dir\") pod \"node-resolver-h69hf\" (UID: \"b53a5c28-a70a-4388-bb82-03cd3832d27a\") " pod="openshift-dns/node-resolver-h69hf" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446470 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-run-multus-certs\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446474 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-etc-openvswitch\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2r67\" (UniqueName: \"kubernetes.io/projected/b53a5c28-a70a-4388-bb82-03cd3832d27a-kube-api-access-q2r67\") pod \"node-resolver-h69hf\" (UID: \"b53a5c28-a70a-4388-bb82-03cd3832d27a\") " pod="openshift-dns/node-resolver-h69hf" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446509 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-run-netns\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.446781 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446514 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-kubelet\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446518 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-run-openvswitch\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-log-socket\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446564 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-sys\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446583 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-sys-fs\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab087428-a2bf-4ca5-8c12-1824860547c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-lib-modules\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-systemd-units\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8g7t\" (UniqueName: \"kubernetes.io/projected/ab087428-a2bf-4ca5-8c12-1824860547c2-kube-api-access-r8g7t\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446643 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-system-cni-dir\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446702 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-kubelet\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-var-lib-cni-bin\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446739 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-var-lib-cni-bin\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-var-lib-cni-multus\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446769 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-sys-fs\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.448584 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-var-lib-kubelet\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446798 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-systemd-units\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc7qf\" (UniqueName: \"kubernetes.io/projected/97edc872-75d9-449b-9fa3-0871dc3a965e-kube-api-access-vc7qf\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446816 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-log-socket\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446845 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-var-lib-kubelet\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-var-lib-openvswitch\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446876 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-host-var-lib-cni-multus\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-cni-netd\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab087428-a2bf-4ca5-8c12-1824860547c2-system-cni-dir\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab087428-a2bf-4ca5-8c12-1824860547c2-os-release\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-multus-cni-dir\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-node-log\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447044 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-cni-netd\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-ovnkube-config\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.446826 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-system-cni-dir\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447116 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab087428-a2bf-4ca5-8c12-1824860547c2-system-cni-dir\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.449419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447119 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-var-lib-openvswitch\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447178 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab087428-a2bf-4ca5-8c12-1824860547c2-os-release\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447233 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-multus-cni-dir\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447269 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-node-log\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447330 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-env-overrides\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2d44\" (UniqueName: \"kubernetes.io/projected/c392372d-4d45-4636-8058-57fd87930e8e-kube-api-access-s2d44\") pod \"network-metrics-daemon-wdvzz\" (UID: \"c392372d-4d45-4636-8058-57fd87930e8e\") " pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/658616f8-26f1-4236-8be0-908fb5650a90-iptables-alerter-script\") pod \"iptables-alerter-lmfzb\" (UID: \"658616f8-26f1-4236-8be0-908fb5650a90\") " pod="openshift-network-operator/iptables-alerter-lmfzb" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/658616f8-26f1-4236-8be0-908fb5650a90-host-slash\") pod \"iptables-alerter-lmfzb\" (UID: \"658616f8-26f1-4236-8be0-908fb5650a90\") " pod="openshift-network-operator/iptables-alerter-lmfzb" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b53a5c28-a70a-4388-bb82-03cd3832d27a-hosts-file\") pod \"node-resolver-h69hf\" (UID: \"b53a5c28-a70a-4388-bb82-03cd3832d27a\") " pod="openshift-dns/node-resolver-h69hf" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447458 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-run-ovn\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab087428-a2bf-4ca5-8c12-1824860547c2-cnibin\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tj2\" (UniqueName: \"kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2\") pod \"network-check-target-bwz4j\" (UID: \"29273a5e-bb56-4d02-9905-33accc2373ef\") " pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-kubernetes\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-etc-selinux\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab087428-a2bf-4ca5-8c12-1824860547c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.450160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4j9\" (UniqueName: \"kubernetes.io/projected/658616f8-26f1-4236-8be0-908fb5650a90-kube-api-access-bx4j9\") pod \"iptables-alerter-lmfzb\" (UID: \"658616f8-26f1-4236-8be0-908fb5650a90\") " pod="openshift-network-operator/iptables-alerter-lmfzb" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4c0c541-bfc7-4376-a46f-622737060480-tmp\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97edc872-75d9-449b-9fa3-0871dc3a965e-multus-daemon-config\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0343e166-1101-45b0-a57e-cc26b4858985-dbus\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447769 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab087428-a2bf-4ca5-8c12-1824860547c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447783 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-sysctl-conf\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447818 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e25aa2e-b99d-492f-abb2-46c3c8fc0662-konnectivity-ca\") pod \"konnectivity-agent-9sj9r\" (UID: \"4e25aa2e-b99d-492f-abb2-46c3c8fc0662\") " pod="kube-system/konnectivity-agent-9sj9r" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447830 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-kubernetes\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447837 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-ovnkube-script-lib\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447857 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b53a5c28-a70a-4388-bb82-03cd3832d27a-hosts-file\") pod \"node-resolver-h69hf\" (UID: \"b53a5c28-a70a-4388-bb82-03cd3832d27a\") " pod="openshift-dns/node-resolver-h69hf" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447884 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-run-ovn\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97e0ac5b-7e47-49c6-a320-6f45f632c3d4-host\") pod \"node-ca-57xjh\" (UID: \"97e0ac5b-7e47-49c6-a320-6f45f632c3d4\") " pod="openshift-image-registry/node-ca-57xjh" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447920 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97e0ac5b-7e47-49c6-a320-6f45f632c3d4-serviceca\") pod \"node-ca-57xjh\" (UID: \"97e0ac5b-7e47-49c6-a320-6f45f632c3d4\") " pod="openshift-image-registry/node-ca-57xjh" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447928 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-ovnkube-config\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab087428-a2bf-4ca5-8c12-1824860547c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447483 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53a5c28-a70a-4388-bb82-03cd3832d27a-tmp-dir\") pod \"node-resolver-h69hf\" (UID: \"b53a5c28-a70a-4388-bb82-03cd3832d27a\") " pod="openshift-dns/node-resolver-h69hf" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447980 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-run\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.451015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4c0c541-bfc7-4376-a46f-622737060480-etc-tuned\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448023 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/658616f8-26f1-4236-8be0-908fb5650a90-host-slash\") pod \"iptables-alerter-lmfzb\" (UID: \"658616f8-26f1-4236-8be0-908fb5650a90\") " pod="openshift-network-operator/iptables-alerter-lmfzb" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-cni-bin\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448061 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab087428-a2bf-4ca5-8c12-1824860547c2-cnibin\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs\") pod \"network-metrics-daemon-wdvzz\" (UID: \"c392372d-4d45-4636-8058-57fd87930e8e\") " pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-registration-dir\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448123 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-device-dir\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-sysctl-d\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-etc-kubernetes\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-socket-dir\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqv8w\" (UniqueName: \"kubernetes.io/projected/8d449419-ae00-4fe7-b11f-3c785ea367fc-kube-api-access-qqv8w\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-run-systemd\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab087428-a2bf-4ca5-8c12-1824860547c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448327 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-sysctl-d\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448372 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-registration-dir\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448410 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-device-dir\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.448596 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:01.451938 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.448664 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs podName:c392372d-4d45-4636-8058-57fd87930e8e nodeName:}" failed. No retries permitted until 2026-07-08 02:58:01.948643077 +0000 UTC m=+2.015831729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs") pod "network-metrics-daemon-wdvzz" (UID: "c392372d-4d45-4636-8058-57fd87930e8e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448701 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-socket-dir\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448749 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-env-overrides\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448802 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab087428-a2bf-4ca5-8c12-1824860547c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448816 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/658616f8-26f1-4236-8be0-908fb5650a90-iptables-alerter-script\") pod \"iptables-alerter-lmfzb\" (UID: \"658616f8-26f1-4236-8be0-908fb5650a90\") " pod="openshift-network-operator/iptables-alerter-lmfzb" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448869 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-run-systemd\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448896 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-etc-sysctl-conf\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.449192 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab087428-a2bf-4ca5-8c12-1824860547c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.449289 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab087428-a2bf-4ca5-8c12-1824860547c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.449283 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-run\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.448599 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97edc872-75d9-449b-9fa3-0871dc3a965e-etc-kubernetes\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.447981 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d449419-ae00-4fe7-b11f-3c785ea367fc-etc-selinux\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.449386 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-host-cni-bin\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.449427 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97edc872-75d9-449b-9fa3-0871dc3a965e-multus-daemon-config\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.449472 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4c0c541-bfc7-4376-a46f-622737060480-lib-modules\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.449911 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97edc872-75d9-449b-9fa3-0871dc3a965e-cni-binary-copy\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.451264 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-ovn-node-metrics-cert\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.451856 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e25aa2e-b99d-492f-abb2-46c3c8fc0662-agent-certs\") pod \"konnectivity-agent-9sj9r\" (UID: \"4e25aa2e-b99d-492f-abb2-46c3c8fc0662\") " pod="kube-system/konnectivity-agent-9sj9r" Jul 08 02:58:01.452877 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.451952 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5j89r" Jul 08 02:58:01.453745 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.452673 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4c0c541-bfc7-4376-a46f-622737060480-tmp\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.453745 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.452708 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4c0c541-bfc7-4376-a46f-622737060480-etc-tuned\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.467180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.467150 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc7qf\" (UniqueName: \"kubernetes.io/projected/97edc872-75d9-449b-9fa3-0871dc3a965e-kube-api-access-vc7qf\") pod \"multus-q2md7\" (UID: \"97edc872-75d9-449b-9fa3-0871dc3a965e\") " pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.467290 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.467202 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98dr2\" (UniqueName: \"kubernetes.io/projected/3bdbbb3c-a277-4cb9-9d63-d626cc52142e-kube-api-access-98dr2\") pod \"ovnkube-node-7qb87\" (UID: \"3bdbbb3c-a277-4cb9-9d63-d626cc52142e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.467290 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.467214 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2d44\" (UniqueName: \"kubernetes.io/projected/c392372d-4d45-4636-8058-57fd87930e8e-kube-api-access-s2d44\") pod \"network-metrics-daemon-wdvzz\" (UID: \"c392372d-4d45-4636-8058-57fd87930e8e\") " pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:01.467494 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.467469 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:58:01.467494 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.467496 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:58:01.467639 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.467511 2576 projected.go:196] Error preparing data for projected volume kube-api-access-q4tj2 for pod openshift-network-diagnostics/network-check-target-bwz4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:01.467639 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.467575 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2 podName:29273a5e-bb56-4d02-9905-33accc2373ef nodeName:}" failed. No retries permitted until 2026-07-08 02:58:01.967557709 +0000 UTC m=+2.034746381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q4tj2" (UniqueName: "kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2") pod "network-check-target-bwz4j" (UID: "29273a5e-bb56-4d02-9905-33accc2373ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:01.468274 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.468254 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4jt2\" (UniqueName: \"kubernetes.io/projected/f4c0c541-bfc7-4376-a46f-622737060480-kube-api-access-b4jt2\") pod \"tuned-27wpf\" (UID: \"f4c0c541-bfc7-4376-a46f-622737060480\") " pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.468516 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.468500 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4j9\" (UniqueName: \"kubernetes.io/projected/658616f8-26f1-4236-8be0-908fb5650a90-kube-api-access-bx4j9\") pod \"iptables-alerter-lmfzb\" (UID: \"658616f8-26f1-4236-8be0-908fb5650a90\") " pod="openshift-network-operator/iptables-alerter-lmfzb" Jul 08 02:58:01.468596 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.468574 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8g7t\" (UniqueName: \"kubernetes.io/projected/ab087428-a2bf-4ca5-8c12-1824860547c2-kube-api-access-r8g7t\") pod \"multus-additional-cni-plugins-nf95s\" (UID: \"ab087428-a2bf-4ca5-8c12-1824860547c2\") " pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.469509 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.469488 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqv8w\" (UniqueName: \"kubernetes.io/projected/8d449419-ae00-4fe7-b11f-3c785ea367fc-kube-api-access-qqv8w\") pod \"aws-ebs-csi-driver-node-ffb92\" (UID: \"8d449419-ae00-4fe7-b11f-3c785ea367fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.469951 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.469925 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2r67\" (UniqueName: \"kubernetes.io/projected/b53a5c28-a70a-4388-bb82-03cd3832d27a-kube-api-access-q2r67\") pod \"node-resolver-h69hf\" (UID: \"b53a5c28-a70a-4388-bb82-03cd3832d27a\") " pod="openshift-dns/node-resolver-h69hf" Jul 08 02:58:01.475137 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.475115 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5j89r" Jul 08 02:58:01.479783 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.479768 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q2md7" Jul 08 02:58:01.497235 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.497207 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:01.509953 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.509930 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" Jul 08 02:58:01.549531 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.549501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97e0ac5b-7e47-49c6-a320-6f45f632c3d4-serviceca\") pod \"node-ca-57xjh\" (UID: \"97e0ac5b-7e47-49c6-a320-6f45f632c3d4\") " pod="openshift-image-registry/node-ca-57xjh" Jul 08 02:58:01.549764 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.549735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0343e166-1101-45b0-a57e-cc26b4858985-kubelet-config\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:01.549906 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.549808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb95t\" (UniqueName: \"kubernetes.io/projected/97e0ac5b-7e47-49c6-a320-6f45f632c3d4-kube-api-access-sb95t\") pod \"node-ca-57xjh\" (UID: \"97e0ac5b-7e47-49c6-a320-6f45f632c3d4\") " pod="openshift-image-registry/node-ca-57xjh" Jul 08 02:58:01.549973 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.549922 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0343e166-1101-45b0-a57e-cc26b4858985-kubelet-config\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:01.550078 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.550051 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97e0ac5b-7e47-49c6-a320-6f45f632c3d4-serviceca\") pod \"node-ca-57xjh\" (UID: \"97e0ac5b-7e47-49c6-a320-6f45f632c3d4\") " pod="openshift-image-registry/node-ca-57xjh" Jul 08 02:58:01.550200 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.550113 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0343e166-1101-45b0-a57e-cc26b4858985-dbus\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:01.550200 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.550136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:01.550200 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.550166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97e0ac5b-7e47-49c6-a320-6f45f632c3d4-host\") pod \"node-ca-57xjh\" (UID: \"97e0ac5b-7e47-49c6-a320-6f45f632c3d4\") " pod="openshift-image-registry/node-ca-57xjh" Jul 08 02:58:01.550385 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.550204 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:01.550385 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.550234 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0343e166-1101-45b0-a57e-cc26b4858985-dbus\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:01.550385 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.550265 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret podName:0343e166-1101-45b0-a57e-cc26b4858985 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:02.050247802 +0000 UTC m=+2.117436444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret") pod "global-pull-secret-syncer-78f5d" (UID: "0343e166-1101-45b0-a57e-cc26b4858985") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:01.550385 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.550340 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97e0ac5b-7e47-49c6-a320-6f45f632c3d4-host\") pod \"node-ca-57xjh\" (UID: \"97e0ac5b-7e47-49c6-a320-6f45f632c3d4\") " pod="openshift-image-registry/node-ca-57xjh" Jul 08 02:58:01.560015 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.559987 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb95t\" (UniqueName: \"kubernetes.io/projected/97e0ac5b-7e47-49c6-a320-6f45f632c3d4-kube-api-access-sb95t\") pod \"node-ca-57xjh\" (UID: \"97e0ac5b-7e47-49c6-a320-6f45f632c3d4\") " pod="openshift-image-registry/node-ca-57xjh" Jul 08 02:58:01.589015 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:01.588968 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8d67d3d75c39b81a69b1a068c27f3b.slice/crio-d865453702b3b6198949c4584d9e5ba602ae02ab0cd1a9015c8a86d73503d08c WatchSource:0}: Error finding container d865453702b3b6198949c4584d9e5ba602ae02ab0cd1a9015c8a86d73503d08c: Status 404 returned error can't find the container with id d865453702b3b6198949c4584d9e5ba602ae02ab0cd1a9015c8a86d73503d08c Jul 08 02:58:01.589360 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:01.589343 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a205a6cae71405e2b5080f71ebefef8.slice/crio-cbc51b6d891ec0cd926c79a951ab7986bca80e88fdf73d9ca89eccaee858a144 WatchSource:0}: Error finding container cbc51b6d891ec0cd926c79a951ab7986bca80e88fdf73d9ca89eccaee858a144: Status 404 returned error can't find the container with id cbc51b6d891ec0cd926c79a951ab7986bca80e88fdf73d9ca89eccaee858a144 Jul 08 02:58:01.593729 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.593712 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 02:58:01.653701 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.653684 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:58:01.655697 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.655677 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h69hf" Jul 08 02:58:01.663062 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:01.663042 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53a5c28_a70a_4388_bb82_03cd3832d27a.slice/crio-fadf7b4ab0b9599df91ae10ebc163b1a88039b0ab23f1a02e0971992f6f4f3ca WatchSource:0}: Error finding container fadf7b4ab0b9599df91ae10ebc163b1a88039b0ab23f1a02e0971992f6f4f3ca: Status 404 returned error can't find the container with id fadf7b4ab0b9599df91ae10ebc163b1a88039b0ab23f1a02e0971992f6f4f3ca Jul 08 02:58:01.669767 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.669749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nf95s" Jul 08 02:58:01.676871 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:01.676847 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab087428_a2bf_4ca5_8c12_1824860547c2.slice/crio-1906d02c305e411e1390acd3166295315c257d9aa8864cf99596a9a57325a434 WatchSource:0}: Error finding container 1906d02c305e411e1390acd3166295315c257d9aa8864cf99596a9a57325a434: Status 404 returned error can't find the container with id 1906d02c305e411e1390acd3166295315c257d9aa8864cf99596a9a57325a434 Jul 08 02:58:01.695807 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.695787 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lmfzb" Jul 08 02:58:01.702911 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:01.702890 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658616f8_26f1_4236_8be0_908fb5650a90.slice/crio-f9a75a9de54bbaf4f84762b76179b1f623769f2e2211aeb7ea2c22fc3b5dead7 WatchSource:0}: Error finding container f9a75a9de54bbaf4f84762b76179b1f623769f2e2211aeb7ea2c22fc3b5dead7: Status 404 returned error can't find the container with id f9a75a9de54bbaf4f84762b76179b1f623769f2e2211aeb7ea2c22fc3b5dead7 Jul 08 02:58:01.714447 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.714432 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9sj9r" Jul 08 02:58:01.720911 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:01.720893 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e25aa2e_b99d_492f_abb2_46c3c8fc0662.slice/crio-6bc06df4272f2c4505186348c0a584a4922fac710315a4d07fe75ec25da816ef WatchSource:0}: Error finding container 6bc06df4272f2c4505186348c0a584a4922fac710315a4d07fe75ec25da816ef: Status 404 returned error can't find the container with id 6bc06df4272f2c4505186348c0a584a4922fac710315a4d07fe75ec25da816ef Jul 08 02:58:01.750041 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.750010 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-27wpf" Jul 08 02:58:01.764769 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:01.764742 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c0c541_bfc7_4376_a46f_622737060480.slice/crio-89627ebb113d5469abb8929b88c33c993f4f498793fb09aa2517be19234e8a2b WatchSource:0}: Error finding container 89627ebb113d5469abb8929b88c33c993f4f498793fb09aa2517be19234e8a2b: Status 404 returned error can't find the container with id 89627ebb113d5469abb8929b88c33c993f4f498793fb09aa2517be19234e8a2b Jul 08 02:58:01.765399 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:01.765379 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bdbbb3c_a277_4cb9_9d63_d626cc52142e.slice/crio-d6c88542d16e764832d2915e680ee867febddc4095f316029678fbcfc5dc8342 WatchSource:0}: Error finding container d6c88542d16e764832d2915e680ee867febddc4095f316029678fbcfc5dc8342: Status 404 returned error can't find the container with id d6c88542d16e764832d2915e680ee867febddc4095f316029678fbcfc5dc8342 Jul 08 02:58:01.815396 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.815360 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-57xjh" Jul 08 02:58:01.822420 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:01.822398 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e0ac5b_7e47_49c6_a320_6f45f632c3d4.slice/crio-e8a669980d40de4865ded75cf5e85d7423806da96770ef0069b7c672e453556a WatchSource:0}: Error finding container e8a669980d40de4865ded75cf5e85d7423806da96770ef0069b7c672e453556a: Status 404 returned error can't find the container with id e8a669980d40de4865ded75cf5e85d7423806da96770ef0069b7c672e453556a Jul 08 02:58:01.864448 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:01.864427 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97edc872_75d9_449b_9fa3_0871dc3a965e.slice/crio-0d8a6615d5148a4311b0632530343a44a2703a185d6f581044f97f6cba46cc8c WatchSource:0}: Error finding container 0d8a6615d5148a4311b0632530343a44a2703a185d6f581044f97f6cba46cc8c: Status 404 returned error can't find the container with id 0d8a6615d5148a4311b0632530343a44a2703a185d6f581044f97f6cba46cc8c Jul 08 02:58:01.885019 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.884992 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:58:01.953715 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:01.953627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs\") pod \"network-metrics-daemon-wdvzz\" (UID: \"c392372d-4d45-4636-8058-57fd87930e8e\") " pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:01.953859 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.953734 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:01.953859 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:01.953789 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs podName:c392372d-4d45-4636-8058-57fd87930e8e nodeName:}" failed. No retries permitted until 2026-07-08 02:58:02.95377579 +0000 UTC m=+3.020964423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs") pod "network-metrics-daemon-wdvzz" (UID: "c392372d-4d45-4636-8058-57fd87930e8e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:02.054534 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:02.054488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tj2\" (UniqueName: \"kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2\") pod \"network-check-target-bwz4j\" (UID: \"29273a5e-bb56-4d02-9905-33accc2373ef\") " pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:02.054694 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:02.054546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:02.054694 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:02.054631 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:58:02.054694 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:02.054654 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:58:02.054694 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:02.054667 2576 projected.go:196] Error preparing data for projected volume kube-api-access-q4tj2 for pod openshift-network-diagnostics/network-check-target-bwz4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:02.054870 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:02.054697 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:02.054870 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:02.054722 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2 podName:29273a5e-bb56-4d02-9905-33accc2373ef nodeName:}" failed. No retries permitted until 2026-07-08 02:58:03.054707649 +0000 UTC m=+3.121896284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q4tj2" (UniqueName: "kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2") pod "network-check-target-bwz4j" (UID: "29273a5e-bb56-4d02-9905-33accc2373ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:02.054870 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:02.054757 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret podName:0343e166-1101-45b0-a57e-cc26b4858985 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:03.054741358 +0000 UTC m=+3.121930007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret") pod "global-pull-secret-syncer-78f5d" (UID: "0343e166-1101-45b0-a57e-cc26b4858985") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:02.087633 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:02.087607 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d449419_ae00_4fe7_b11f_3c785ea367fc.slice/crio-387c1de697ec018364ff9dbcd21b8092606743e8de1202193dc8752e024562f4 WatchSource:0}: Error finding container 387c1de697ec018364ff9dbcd21b8092606743e8de1202193dc8752e024562f4: Status 404 returned error can't find the container with id 387c1de697ec018364ff9dbcd21b8092606743e8de1202193dc8752e024562f4 Jul 08 02:58:03.523225 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.522525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:03.523225 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:03.522659 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:03.533514 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.533204 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-27wpf" event={"ID":"f4c0c541-bfc7-4376-a46f-622737060480","Type":"ContainerStarted","Data":"89627ebb113d5469abb8929b88c33c993f4f498793fb09aa2517be19234e8a2b"} Jul 08 02:58:03.542744 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.542611 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9sj9r" event={"ID":"4e25aa2e-b99d-492f-abb2-46c3c8fc0662","Type":"ContainerStarted","Data":"6bc06df4272f2c4505186348c0a584a4922fac710315a4d07fe75ec25da816ef"} Jul 08 02:58:03.547546 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.547478 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lmfzb" event={"ID":"658616f8-26f1-4236-8be0-908fb5650a90","Type":"ContainerStarted","Data":"f9a75a9de54bbaf4f84762b76179b1f623769f2e2211aeb7ea2c22fc3b5dead7"} Jul 08 02:58:03.554526 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.554421 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 02:53:02 +0000 UTC" deadline="2028-04-24 11:55:16.6048914 +0000 UTC" Jul 08 02:58:03.554526 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.554457 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15752h57m13.050439257s" Jul 08 02:58:03.557445 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.557360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf95s" event={"ID":"ab087428-a2bf-4ca5-8c12-1824860547c2","Type":"ContainerStarted","Data":"1906d02c305e411e1390acd3166295315c257d9aa8864cf99596a9a57325a434"} Jul 08 02:58:03.577113 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.576995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" event={"ID":"3bdbbb3c-a277-4cb9-9d63-d626cc52142e","Type":"ContainerStarted","Data":"d6c88542d16e764832d2915e680ee867febddc4095f316029678fbcfc5dc8342"} Jul 08 02:58:03.583728 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.583690 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h69hf" event={"ID":"b53a5c28-a70a-4388-bb82-03cd3832d27a","Type":"ContainerStarted","Data":"fadf7b4ab0b9599df91ae10ebc163b1a88039b0ab23f1a02e0971992f6f4f3ca"} Jul 08 02:58:03.594136 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.594033 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" event={"ID":"1a205a6cae71405e2b5080f71ebefef8","Type":"ContainerStarted","Data":"cbc51b6d891ec0cd926c79a951ab7986bca80e88fdf73d9ca89eccaee858a144"} Jul 08 02:58:03.614850 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.614821 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal" event={"ID":"8b8d67d3d75c39b81a69b1a068c27f3b","Type":"ContainerStarted","Data":"d865453702b3b6198949c4584d9e5ba602ae02ab0cd1a9015c8a86d73503d08c"} Jul 08 02:58:03.622394 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.622367 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" event={"ID":"8d449419-ae00-4fe7-b11f-3c785ea367fc","Type":"ContainerStarted","Data":"387c1de697ec018364ff9dbcd21b8092606743e8de1202193dc8752e024562f4"} Jul 08 02:58:03.626577 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.626556 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q2md7" event={"ID":"97edc872-75d9-449b-9fa3-0871dc3a965e","Type":"ContainerStarted","Data":"0d8a6615d5148a4311b0632530343a44a2703a185d6f581044f97f6cba46cc8c"} Jul 08 02:58:03.635473 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.635446 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-57xjh" event={"ID":"97e0ac5b-7e47-49c6-a320-6f45f632c3d4","Type":"ContainerStarted","Data":"e8a669980d40de4865ded75cf5e85d7423806da96770ef0069b7c672e453556a"} Jul 08 02:58:03.827149 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:03.825583 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:58:04.039587 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:04.039354 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs\") pod \"network-metrics-daemon-wdvzz\" (UID: \"c392372d-4d45-4636-8058-57fd87930e8e\") " pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:04.039587 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:04.039504 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:04.039587 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:04.039565 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs podName:c392372d-4d45-4636-8058-57fd87930e8e nodeName:}" failed. No retries permitted until 2026-07-08 02:58:06.039546399 +0000 UTC m=+5.029994428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs") pod "network-metrics-daemon-wdvzz" (UID: "c392372d-4d45-4636-8058-57fd87930e8e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:04.140540 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:04.140457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:04.140709 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:04.140564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tj2\" (UniqueName: \"kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2\") pod \"network-check-target-bwz4j\" (UID: \"29273a5e-bb56-4d02-9905-33accc2373ef\") " pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:04.140783 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:04.140706 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:58:04.140783 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:04.140729 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:58:04.140783 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:04.140743 2576 projected.go:196] Error preparing data for projected volume kube-api-access-q4tj2 for pod openshift-network-diagnostics/network-check-target-bwz4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:04.140944 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:04.140801 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2 podName:29273a5e-bb56-4d02-9905-33accc2373ef nodeName:}" failed. No retries permitted until 2026-07-08 02:58:06.140783445 +0000 UTC m=+5.131231477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q4tj2" (UniqueName: "kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2") pod "network-check-target-bwz4j" (UID: "29273a5e-bb56-4d02-9905-33accc2373ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:04.141233 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:04.141213 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:04.141317 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:04.141277 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret podName:0343e166-1101-45b0-a57e-cc26b4858985 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:06.14125892 +0000 UTC m=+5.131706941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret") pod "global-pull-secret-syncer-78f5d" (UID: "0343e166-1101-45b0-a57e-cc26b4858985") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:04.520457 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:04.519711 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:04.520457 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:04.519851 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:04.520457 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:04.520300 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:04.520457 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:04.520409 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:04.555514 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:04.555468 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 02:53:02 +0000 UTC" deadline="2028-03-14 03:40:14.373708468 +0000 UTC" Jul 08 02:58:04.555514 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:04.555511 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14760h42m9.81820287s" Jul 08 02:58:05.520450 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:05.520401 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:05.520631 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:05.520539 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:06.055819 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:06.055776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs\") pod \"network-metrics-daemon-wdvzz\" (UID: \"c392372d-4d45-4636-8058-57fd87930e8e\") " pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:06.056308 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:06.055930 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:06.056308 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:06.055994 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs podName:c392372d-4d45-4636-8058-57fd87930e8e nodeName:}" failed. No retries permitted until 2026-07-08 02:58:10.055973206 +0000 UTC m=+9.046421225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs") pod "network-metrics-daemon-wdvzz" (UID: "c392372d-4d45-4636-8058-57fd87930e8e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:06.156289 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:06.156245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tj2\" (UniqueName: \"kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2\") pod \"network-check-target-bwz4j\" (UID: \"29273a5e-bb56-4d02-9905-33accc2373ef\") " pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:06.156471 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:06.156299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:06.156471 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:06.156442 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:06.156582 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:06.156511 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret podName:0343e166-1101-45b0-a57e-cc26b4858985 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:10.156483869 +0000 UTC m=+9.146931886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret") pod "global-pull-secret-syncer-78f5d" (UID: "0343e166-1101-45b0-a57e-cc26b4858985") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:06.156916 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:06.156899 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:58:06.156982 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:06.156925 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:58:06.156982 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:06.156938 2576 projected.go:196] Error preparing data for projected volume kube-api-access-q4tj2 for pod openshift-network-diagnostics/network-check-target-bwz4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:06.156982 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:06.156979 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2 podName:29273a5e-bb56-4d02-9905-33accc2373ef nodeName:}" failed. No retries permitted until 2026-07-08 02:58:10.156965539 +0000 UTC m=+9.147413559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q4tj2" (UniqueName: "kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2") pod "network-check-target-bwz4j" (UID: "29273a5e-bb56-4d02-9905-33accc2373ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:06.520466 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:06.519957 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:06.520466 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:06.519965 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:06.520466 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:06.520131 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:06.520466 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:06.520314 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:07.520293 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:07.520258 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:07.520719 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:07.520394 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:08.519721 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:08.519688 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:08.519926 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:08.519828 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:08.520153 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:08.520133 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:08.520264 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:08.520246 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:09.519873 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:09.519836 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:09.520359 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:09.519954 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:10.088633 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:10.088000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs\") pod \"network-metrics-daemon-wdvzz\" (UID: \"c392372d-4d45-4636-8058-57fd87930e8e\") " pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:10.088633 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:10.088171 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:10.088633 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:10.088252 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs podName:c392372d-4d45-4636-8058-57fd87930e8e nodeName:}" failed. No retries permitted until 2026-07-08 02:58:18.088229401 +0000 UTC m=+17.078677430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs") pod "network-metrics-daemon-wdvzz" (UID: "c392372d-4d45-4636-8058-57fd87930e8e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:10.188767 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:10.188725 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tj2\" (UniqueName: \"kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2\") pod \"network-check-target-bwz4j\" (UID: \"29273a5e-bb56-4d02-9905-33accc2373ef\") " pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:10.188911 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:10.188783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:10.188955 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:10.188919 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:10.188988 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:10.188978 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret podName:0343e166-1101-45b0-a57e-cc26b4858985 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:18.188961717 +0000 UTC m=+17.179409736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret") pod "global-pull-secret-syncer-78f5d" (UID: "0343e166-1101-45b0-a57e-cc26b4858985") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:10.189514 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:10.189410 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:58:10.189514 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:10.189433 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:58:10.189514 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:10.189442 2576 projected.go:196] Error preparing data for projected volume kube-api-access-q4tj2 for pod openshift-network-diagnostics/network-check-target-bwz4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:10.189514 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:10.189483 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2 podName:29273a5e-bb56-4d02-9905-33accc2373ef nodeName:}" failed. No retries permitted until 2026-07-08 02:58:18.189467502 +0000 UTC m=+17.179915531 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q4tj2" (UniqueName: "kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2") pod "network-check-target-bwz4j" (UID: "29273a5e-bb56-4d02-9905-33accc2373ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:10.519672 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:10.519634 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:10.519859 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:10.519769 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:10.520292 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:10.520169 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:10.520292 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:10.520248 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:11.520199 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:11.520155 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:11.520402 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:11.520283 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:12.520336 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:12.520298 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:12.520542 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:12.520452 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:12.520542 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:12.520526 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:12.520955 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:12.520600 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:13.520297 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:13.519806 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:13.520297 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:13.519939 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:14.519760 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:14.519726 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:14.520159 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:14.519882 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:14.520159 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:14.519946 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:14.520159 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:14.520107 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:15.520182 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:15.520140 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:15.520719 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:15.520283 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:16.519724 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:16.519691 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:16.519724 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:16.519707 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:16.519922 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:16.519806 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:16.519922 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:16.519897 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:17.519543 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:17.519505 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:17.519977 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:17.519635 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:18.149447 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:18.149407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs\") pod \"network-metrics-daemon-wdvzz\" (UID: \"c392372d-4d45-4636-8058-57fd87930e8e\") " pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:18.149643 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:18.149562 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:18.149643 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:18.149613 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs podName:c392372d-4d45-4636-8058-57fd87930e8e nodeName:}" failed. No retries permitted until 2026-07-08 02:58:34.149598174 +0000 UTC m=+33.140046193 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs") pod "network-metrics-daemon-wdvzz" (UID: "c392372d-4d45-4636-8058-57fd87930e8e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:18.250554 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:18.250517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tj2\" (UniqueName: \"kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2\") pod \"network-check-target-bwz4j\" (UID: \"29273a5e-bb56-4d02-9905-33accc2373ef\") " pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:18.250708 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:18.250568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:18.250708 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:18.250637 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:58:18.250708 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:18.250656 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:58:18.250708 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:18.250666 2576 projected.go:196] Error preparing data for projected volume kube-api-access-q4tj2 for pod openshift-network-diagnostics/network-check-target-bwz4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:18.250708 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:18.250675 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:18.250863 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:18.250715 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2 podName:29273a5e-bb56-4d02-9905-33accc2373ef nodeName:}" failed. No retries permitted until 2026-07-08 02:58:34.250699414 +0000 UTC m=+33.241147448 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q4tj2" (UniqueName: "kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2") pod "network-check-target-bwz4j" (UID: "29273a5e-bb56-4d02-9905-33accc2373ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:18.250863 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:18.250729 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret podName:0343e166-1101-45b0-a57e-cc26b4858985 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:34.250723246 +0000 UTC m=+33.241171260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret") pod "global-pull-secret-syncer-78f5d" (UID: "0343e166-1101-45b0-a57e-cc26b4858985") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:18.519625 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:18.519595 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:18.519970 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:18.519595 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:18.519970 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:18.519695 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:18.519970 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:18.519764 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:19.519577 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:19.519536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:19.519748 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:19.519683 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:20.519774 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:20.519736 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:20.520186 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:20.519749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:20.520186 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:20.519872 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:20.520186 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:20.519936 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:21.520905 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:21.520874 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:21.521356 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:21.520995 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:22.519610 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:22.519576 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:22.519610 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:22.519596 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:22.519852 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:22.519698 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:22.519852 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:22.519788 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:23.520026 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:23.519840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:23.520521 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:23.520146 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:23.701862 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:23.701828 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q2md7" event={"ID":"97edc872-75d9-449b-9fa3-0871dc3a965e","Type":"ContainerStarted","Data":"b276fa2b7c1169f9f73d7dff20ba8f68617a3d8e23014af84e5b71d70b8b9a46"} Jul 08 02:58:23.703401 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:23.703373 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-27wpf" event={"ID":"f4c0c541-bfc7-4376-a46f-622737060480","Type":"ContainerStarted","Data":"d82da4cd3ff23159e0405c2dc4ef8ab76c591bc788d299901150ff259544551b"} Jul 08 02:58:23.707666 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:23.707649 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 02:58:23.708213 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:23.708188 2576 generic.go:356] "Generic (PLEG): container finished" podID="3bdbbb3c-a277-4cb9-9d63-d626cc52142e" containerID="4ca3c8a1da60b955cc35c5ca0450847dea03f4498e8b622da54b4bb80fceb3de" exitCode=1 Jul 08 02:58:23.708313 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:23.708216 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" event={"ID":"3bdbbb3c-a277-4cb9-9d63-d626cc52142e","Type":"ContainerStarted","Data":"d3b53525372af3579a0c9cdacfe0c47ce1de92bfaa85e03c0c3ada7f13349103"} Jul 08 02:58:23.708313 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:23.708274 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" event={"ID":"3bdbbb3c-a277-4cb9-9d63-d626cc52142e","Type":"ContainerStarted","Data":"b6ceff246ab51d131e9acfb8ce90e902d928ff57baa6a5f2f2302e29c1dc03b7"} Jul 08 02:58:23.708313 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:23.708312 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" event={"ID":"3bdbbb3c-a277-4cb9-9d63-d626cc52142e","Type":"ContainerStarted","Data":"17f808250b4e048d23e4b41b54367092be94d05dbab9c910319ffb17c45eb86a"} Jul 08 02:58:23.708426 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:23.708343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" event={"ID":"3bdbbb3c-a277-4cb9-9d63-d626cc52142e","Type":"ContainerDied","Data":"4ca3c8a1da60b955cc35c5ca0450847dea03f4498e8b622da54b4bb80fceb3de"} Jul 08 02:58:23.708426 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:23.708377 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" event={"ID":"3bdbbb3c-a277-4cb9-9d63-d626cc52142e","Type":"ContainerStarted","Data":"0ecb9c4c54f496a919789f8c0232b9fc02d233f8dd4334d2942163d00a957bb3"} Jul 08 02:58:23.709847 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:23.709827 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal" event={"ID":"8b8d67d3d75c39b81a69b1a068c27f3b","Type":"ContainerStarted","Data":"ff364e84f03393c44b81a73cb81e989ce83ab86882a9d15c2e4c42fa9284b2fb"} Jul 08 02:58:23.721510 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:23.721424 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-q2md7" podStartSLOduration=2.628457885 podStartE2EDuration="22.721410656s" totalImagesPullingTime="20.092952771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:23.721261511 +0000 UTC m=+22.711709548" watchObservedRunningTime="2026-07-08 02:58:23.721410656 +0000 UTC m=+22.711858692" Jul 08 02:58:23.741774 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:23.741739 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal" podStartSLOduration=21.741726653 podStartE2EDuration="21.741726653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:23.741540814 +0000 UTC m=+22.731988853" watchObservedRunningTime="2026-07-08 02:58:23.741726653 +0000 UTC m=+22.732174689" Jul 08 02:58:23.761349 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:23.761312 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-27wpf" podStartSLOduration=2.6286834199999998 podStartE2EDuration="22.761300707s" totalImagesPullingTime="20.132617287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:23.760859578 +0000 UTC m=+22.751307627" watchObservedRunningTime="2026-07-08 02:58:23.761300707 +0000 UTC m=+22.751748743" Jul 08 02:58:24.520005 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.519830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:24.520189 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.519885 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:24.520189 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:24.520117 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:24.520189 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:24.520157 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:24.713793 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.713759 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-57xjh" event={"ID":"97e0ac5b-7e47-49c6-a320-6f45f632c3d4","Type":"ContainerStarted","Data":"f63a7f4b721d4c7ca5332962ec827ced5cda15df5673e0dcefb0ed77a55263f6"} Jul 08 02:58:24.715273 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.715245 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9sj9r" event={"ID":"4e25aa2e-b99d-492f-abb2-46c3c8fc0662","Type":"ContainerStarted","Data":"fc621b178e473d6019c373907246446b54ba72ef8fbbb99e13f49f017ca822c0"} Jul 08 02:58:24.716704 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.716684 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lmfzb" event={"ID":"658616f8-26f1-4236-8be0-908fb5650a90","Type":"ContainerStarted","Data":"4c7845b8c889e4fe4f854d90d585fb408032983eda3f3fbd006aea83d770669e"} Jul 08 02:58:24.718176 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.718154 2576 generic.go:356] "Generic (PLEG): container finished" podID="ab087428-a2bf-4ca5-8c12-1824860547c2" containerID="a25d989a731ba477b63961f199be1effe548de442766a97addc36c068718acc0" exitCode=0 Jul 08 02:58:24.718262 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.718227 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf95s" event={"ID":"ab087428-a2bf-4ca5-8c12-1824860547c2","Type":"ContainerDied","Data":"a25d989a731ba477b63961f199be1effe548de442766a97addc36c068718acc0"} Jul 08 02:58:24.723447 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.723431 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 02:58:24.723793 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.723774 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" event={"ID":"3bdbbb3c-a277-4cb9-9d63-d626cc52142e","Type":"ContainerStarted","Data":"27842b86e9c9151f291f6165d6ee141392d25ef9e4c1456a09e6603ef8ed4a00"} Jul 08 02:58:24.725187 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.725163 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h69hf" event={"ID":"b53a5c28-a70a-4388-bb82-03cd3832d27a","Type":"ContainerStarted","Data":"d9ef983c93bbb8dc3bb048b055d973fac6fa4c3ef5f5adff444a532573a2f4d9"} Jul 08 02:58:24.726539 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.726518 2576 generic.go:356] "Generic (PLEG): container finished" podID="1a205a6cae71405e2b5080f71ebefef8" containerID="d490119b185e728c410ebd41902f8a3a420d94437df63e669a184a23748c0daf" exitCode=0 Jul 08 02:58:24.726626 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.726601 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" event={"ID":"1a205a6cae71405e2b5080f71ebefef8","Type":"ContainerDied","Data":"d490119b185e728c410ebd41902f8a3a420d94437df63e669a184a23748c0daf"} Jul 08 02:58:24.728115 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.728094 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" event={"ID":"8d449419-ae00-4fe7-b11f-3c785ea367fc","Type":"ContainerStarted","Data":"54d0700e1742007721a8a8749f9289076ba6e2468a87e3d9cffc375e8e4f1f7b"} Jul 08 02:58:24.728575 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.728540 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-57xjh" podStartSLOduration=2.648739633 podStartE2EDuration="22.728527411s" totalImagesPullingTime="20.079787778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:24.728366763 +0000 UTC m=+23.718814801" watchObservedRunningTime="2026-07-08 02:58:24.728527411 +0000 UTC m=+23.718975448" Jul 08 02:58:24.742675 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.742643 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9sj9r" podStartSLOduration=3.56058573 podStartE2EDuration="23.742628323s" totalImagesPullingTime="20.182042593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:24.742527821 +0000 UTC m=+23.732975859" watchObservedRunningTime="2026-07-08 02:58:24.742628323 +0000 UTC m=+23.733076361" Jul 08 02:58:24.773902 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.773834 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h69hf" podStartSLOduration=3.5361130960000002 podStartE2EDuration="23.773820243s" totalImagesPullingTime="20.237707147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:24.773568959 +0000 UTC m=+23.764016996" watchObservedRunningTime="2026-07-08 02:58:24.773820243 +0000 UTC m=+23.764268284" Jul 08 02:58:24.809721 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:24.807999 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lmfzb" podStartSLOduration=3.608948207 podStartE2EDuration="23.807951912s" totalImagesPullingTime="20.199003705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:24.806444627 +0000 UTC m=+23.796892661" watchObservedRunningTime="2026-07-08 02:58:24.807951912 +0000 UTC m=+23.798399950" Jul 08 02:58:25.520176 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:25.520136 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:25.520866 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:25.520263 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:25.731766 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:25.731728 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" event={"ID":"1a205a6cae71405e2b5080f71ebefef8","Type":"ContainerStarted","Data":"017236c0e5ea99f58a3d80cbfaf801fdfcd4836f90f200058ada5965c01b65b0"} Jul 08 02:58:25.747034 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:25.746983 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" podStartSLOduration=23.746967467 podStartE2EDuration="23.746967467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:25.74670926 +0000 UTC m=+24.737157299" watchObservedRunningTime="2026-07-08 02:58:25.746967467 +0000 UTC m=+24.737415503" Jul 08 02:58:25.803179 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:25.803151 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 02:58:26.491616 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:26.491516 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T02:58:25.803174074Z","UUID":"5f11cbcf-4636-4a58-b03e-b1ed6218641d","Handler":null,"Name":"","Endpoint":""} Jul 08 02:58:26.493282 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:26.493254 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 02:58:26.493415 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:26.493301 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 02:58:26.519625 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:26.519600 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:26.519734 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:26.519607 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:26.519734 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:26.519701 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:26.519837 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:26.519778 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:26.736553 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:26.736467 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" event={"ID":"8d449419-ae00-4fe7-b11f-3c785ea367fc","Type":"ContainerStarted","Data":"9981cbf8e114d5f208ce91a18316ea02fd1548d2b07c448d6b1883df47fe922e"} Jul 08 02:58:26.740814 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:26.740780 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 02:58:26.741697 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:26.741637 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" event={"ID":"3bdbbb3c-a277-4cb9-9d63-d626cc52142e","Type":"ContainerStarted","Data":"3be1f427afa33b7abab3fc6c62073b2dfbbebc3f9374afe14ca4809de46f2b8d"} Jul 08 02:58:27.432804 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:27.432782 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9sj9r" Jul 08 02:58:27.433457 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:27.433442 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9sj9r" Jul 08 02:58:27.520031 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:27.520002 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:27.520197 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:27.520142 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:27.745377 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:27.745295 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" event={"ID":"8d449419-ae00-4fe7-b11f-3c785ea367fc","Type":"ContainerStarted","Data":"82f62e341712fc533d10a6fd61aeaca485c55ebe6d2c34fb10dc20455852843f"} Jul 08 02:58:27.767735 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:27.767681 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffb92" podStartSLOduration=3.075150561 podStartE2EDuration="25.767667171s" totalImagesPullingTime="22.69251661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:27.767456047 +0000 UTC m=+26.757904084" watchObservedRunningTime="2026-07-08 02:58:27.767667171 +0000 UTC m=+26.758115208" Jul 08 02:58:28.519815 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:28.519634 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:28.519994 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:28.519639 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:28.519994 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:28.519889 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:28.520138 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:28.519993 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:28.752005 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:28.751972 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 02:58:28.752874 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:28.752557 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" event={"ID":"3bdbbb3c-a277-4cb9-9d63-d626cc52142e","Type":"ContainerStarted","Data":"b4c407c47f5782ea4db92209a6a66bb04b7026d7bc07d259f1e9d9e89fc9994b"} Jul 08 02:58:28.752989 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:28.752973 2576 scope.go:117] "RemoveContainer" containerID="4ca3c8a1da60b955cc35c5ca0450847dea03f4498e8b622da54b4bb80fceb3de" Jul 08 02:58:29.519694 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:29.519661 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:29.519887 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:29.519804 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:29.761037 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:29.761013 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 02:58:29.761447 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:29.761422 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" event={"ID":"3bdbbb3c-a277-4cb9-9d63-d626cc52142e","Type":"ContainerStarted","Data":"6e2863a400292540987f1a50cd055276031ff0af673c5957853e858efab66dd3"} Jul 08 02:58:29.762573 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:29.762549 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:29.762688 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:29.762649 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:29.762768 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:29.762758 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:29.782098 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:29.781957 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:29.784357 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:29.784335 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:58:29.791247 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:29.791191 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" podStartSLOduration=8.588779446 podStartE2EDuration="28.791175725s" totalImagesPullingTime="20.202396279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:29.789509794 +0000 UTC m=+28.779957844" watchObservedRunningTime="2026-07-08 02:58:29.791175725 +0000 UTC m=+28.781623762" Jul 08 02:58:30.032778 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:30.032677 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-78f5d"] Jul 08 02:58:30.032913 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:30.032840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:30.032962 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:30.032929 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:30.036440 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:30.036412 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wdvzz"] Jul 08 02:58:30.036602 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:30.036536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:30.036662 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:30.036645 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:30.037375 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:30.037354 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bwz4j"] Jul 08 02:58:30.037510 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:30.037425 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:30.037510 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:30.037477 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:31.520325 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:31.520295 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:31.520798 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:31.520392 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:31.520798 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:31.520459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:31.520798 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:31.520583 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:31.520798 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:31.520620 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:31.520798 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:31.520677 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:31.768345 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:31.768296 2576 generic.go:356] "Generic (PLEG): container finished" podID="ab087428-a2bf-4ca5-8c12-1824860547c2" containerID="31b82419957cbccb8764aaf605cd186ca4533153ab2156a4feda5904b3f100f0" exitCode=0 Jul 08 02:58:31.768483 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:31.768380 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf95s" event={"ID":"ab087428-a2bf-4ca5-8c12-1824860547c2","Type":"ContainerDied","Data":"31b82419957cbccb8764aaf605cd186ca4533153ab2156a4feda5904b3f100f0"} Jul 08 02:58:32.705890 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:32.705754 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9sj9r" Jul 08 02:58:32.706270 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:32.706002 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 08 02:58:32.706527 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:32.706508 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9sj9r" Jul 08 02:58:32.772804 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:32.772779 2576 generic.go:356] "Generic (PLEG): container finished" podID="ab087428-a2bf-4ca5-8c12-1824860547c2" containerID="21c309fdea27c61646f801fa28966b9339a9104f4a958cfa1d2095343867cd69" exitCode=0 Jul 08 02:58:32.772911 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:32.772860 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf95s" event={"ID":"ab087428-a2bf-4ca5-8c12-1824860547c2","Type":"ContainerDied","Data":"21c309fdea27c61646f801fa28966b9339a9104f4a958cfa1d2095343867cd69"} Jul 08 02:58:33.519977 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:33.519945 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:33.520160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:33.519945 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:33.520160 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:33.520080 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:33.520160 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:33.519945 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:33.520337 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:33.520167 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:33.520337 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:33.520246 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:33.778022 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:33.777934 2576 generic.go:356] "Generic (PLEG): container finished" podID="ab087428-a2bf-4ca5-8c12-1824860547c2" containerID="ba99cac3b0a0c1d3e12577c7523c5309b27fdca8a0dcff535735825a4d54c4b8" exitCode=0 Jul 08 02:58:33.778022 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:33.777992 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf95s" event={"ID":"ab087428-a2bf-4ca5-8c12-1824860547c2","Type":"ContainerDied","Data":"ba99cac3b0a0c1d3e12577c7523c5309b27fdca8a0dcff535735825a4d54c4b8"} Jul 08 02:58:34.171694 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:34.171666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs\") pod \"network-metrics-daemon-wdvzz\" (UID: \"c392372d-4d45-4636-8058-57fd87930e8e\") " pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:34.171828 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:34.171765 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:34.171828 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:34.171819 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs podName:c392372d-4d45-4636-8058-57fd87930e8e nodeName:}" failed. No retries permitted until 2026-07-08 02:59:06.171806272 +0000 UTC m=+65.162254291 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs") pod "network-metrics-daemon-wdvzz" (UID: "c392372d-4d45-4636-8058-57fd87930e8e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:34.272484 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:34.272444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tj2\" (UniqueName: \"kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2\") pod \"network-check-target-bwz4j\" (UID: \"29273a5e-bb56-4d02-9905-33accc2373ef\") " pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:34.272484 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:34.272487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:34.272748 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:34.272600 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:34.272748 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:34.272643 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:58:34.272748 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:34.272672 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:58:34.272748 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:34.272689 2576 projected.go:196] Error preparing data for projected volume kube-api-access-q4tj2 for pod openshift-network-diagnostics/network-check-target-bwz4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:34.272748 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:34.272651 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret podName:0343e166-1101-45b0-a57e-cc26b4858985 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:06.272638642 +0000 UTC m=+65.263086657 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret") pod "global-pull-secret-syncer-78f5d" (UID: "0343e166-1101-45b0-a57e-cc26b4858985") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:34.272915 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:34.272762 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2 podName:29273a5e-bb56-4d02-9905-33accc2373ef nodeName:}" failed. No retries permitted until 2026-07-08 02:59:06.272742847 +0000 UTC m=+65.263190868 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q4tj2" (UniqueName: "kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2") pod "network-check-target-bwz4j" (UID: "29273a5e-bb56-4d02-9905-33accc2373ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:35.520371 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.520339 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:35.520934 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.520339 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:35.520934 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:35.520474 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwz4j" podUID="29273a5e-bb56-4d02-9905-33accc2373ef" Jul 08 02:58:35.520934 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.520487 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:35.520934 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:35.520601 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdvzz" podUID="c392372d-4d45-4636-8058-57fd87930e8e" Jul 08 02:58:35.520934 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:35.520689 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78f5d" podUID="0343e166-1101-45b0-a57e-cc26b4858985" Jul 08 02:58:35.836466 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.836386 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeReady" Jul 08 02:58:35.836603 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.836530 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 02:58:35.890289 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.890253 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c4cc5dff5-lx8g9"] Jul 08 02:58:35.892357 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.892335 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:35.895962 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.895658 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 08 02:58:35.895962 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.895685 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xcxgk\"" Jul 08 02:58:35.895962 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.895729 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 08 02:58:35.895962 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.895761 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 08 02:58:35.900969 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.900945 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 08 02:58:35.923322 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.923297 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c4cc5dff5-lx8g9"] Jul 08 02:58:35.930419 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.930388 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6j82n"] Jul 08 02:58:35.932180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.932162 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lbwvc"] Jul 08 02:58:35.932312 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.932294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:35.933938 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.933913 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lbwvc" Jul 08 02:58:35.935961 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.935942 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 02:58:35.936225 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.936209 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 02:58:35.940807 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.940785 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kmjwn\"" Jul 08 02:58:35.940965 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.940950 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 02:58:35.941334 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.941311 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6djh2\"" Jul 08 02:58:35.941432 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.941392 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 02:58:35.941695 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.941670 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 02:58:35.952393 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.952365 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6j82n"] Jul 08 02:58:35.953973 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.953954 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lbwvc"] Jul 08 02:58:35.985371 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.985341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-certificates\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:35.985519 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.985384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-installation-pull-secrets\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:35.985519 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.985415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-bound-sa-token\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:35.985519 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.985460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdn9k\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-kube-api-access-qdn9k\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:35.985621 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.985519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-image-registry-private-configuration\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:35.985621 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.985548 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-ca-trust-extracted\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:35.985621 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.985572 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-trusted-ca\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:35.985621 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:35.985607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.086115 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c47594-a484-4654-8cae-7325c0d9d581-tmp-dir\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:36.086115 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.086361 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03-cert\") pod \"ingress-canary-lbwvc\" (UID: \"cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03\") " pod="openshift-ingress-canary/ingress-canary-lbwvc" Jul 08 02:58:36.086361 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-certificates\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.086361 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:36.086261 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 02:58:36.086361 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:36.086283 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c4cc5dff5-lx8g9: secret "image-registry-tls" not found Jul 08 02:58:36.086361 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:36.086344 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls podName:80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:36.586322649 +0000 UTC m=+35.576770664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls") pod "image-registry-6c4cc5dff5-lx8g9" (UID: "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5") : secret "image-registry-tls" not found Jul 08 02:58:36.086616 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-installation-pull-secrets\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.086616 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-bound-sa-token\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.086616 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj7fm\" (UniqueName: \"kubernetes.io/projected/94c47594-a484-4654-8cae-7325c0d9d581-kube-api-access-nj7fm\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:36.086616 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdn9k\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-kube-api-access-qdn9k\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.086616 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086550 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94c47594-a484-4654-8cae-7325c0d9d581-config-volume\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:36.086830 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:36.086830 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-image-registry-private-configuration\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.086830 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-ca-trust-extracted\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.086830 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-trusted-ca\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.086830 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7p52\" (UniqueName: \"kubernetes.io/projected/cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03-kube-api-access-h7p52\") pod \"ingress-canary-lbwvc\" (UID: \"cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03\") " pod="openshift-ingress-canary/ingress-canary-lbwvc" Jul 08 02:58:36.087074 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.086944 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-certificates\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.087114 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.087086 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-ca-trust-extracted\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.087549 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.087530 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-trusted-ca\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.090348 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.090322 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-image-registry-private-configuration\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.090437 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.090330 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-installation-pull-secrets\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.095617 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.095589 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdn9k\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-kube-api-access-qdn9k\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.095749 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.095731 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-bound-sa-token\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.187421 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.187383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03-cert\") pod \"ingress-canary-lbwvc\" (UID: \"cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03\") " pod="openshift-ingress-canary/ingress-canary-lbwvc" Jul 08 02:58:36.187580 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.187454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj7fm\" (UniqueName: \"kubernetes.io/projected/94c47594-a484-4654-8cae-7325c0d9d581-kube-api-access-nj7fm\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:36.187580 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.187483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94c47594-a484-4654-8cae-7325c0d9d581-config-volume\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:36.187580 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.187513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:36.187736 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.187598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7p52\" (UniqueName: \"kubernetes.io/projected/cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03-kube-api-access-h7p52\") pod \"ingress-canary-lbwvc\" (UID: \"cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03\") " pod="openshift-ingress-canary/ingress-canary-lbwvc" Jul 08 02:58:36.187736 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.187643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c47594-a484-4654-8cae-7325c0d9d581-tmp-dir\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:36.187828 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:36.187768 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:58:36.187876 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:36.187841 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls podName:94c47594-a484-4654-8cae-7325c0d9d581 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:36.687819983 +0000 UTC m=+35.678268011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls") pod "dns-default-6j82n" (UID: "94c47594-a484-4654-8cae-7325c0d9d581") : secret "dns-default-metrics-tls" not found Jul 08 02:58:36.187956 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.187935 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c47594-a484-4654-8cae-7325c0d9d581-tmp-dir\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:36.188207 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.188184 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94c47594-a484-4654-8cae-7325c0d9d581-config-volume\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:36.190200 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.190170 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03-cert\") pod \"ingress-canary-lbwvc\" (UID: \"cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03\") " pod="openshift-ingress-canary/ingress-canary-lbwvc" Jul 08 02:58:36.206192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.206173 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7p52\" (UniqueName: \"kubernetes.io/projected/cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03-kube-api-access-h7p52\") pod \"ingress-canary-lbwvc\" (UID: \"cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03\") " pod="openshift-ingress-canary/ingress-canary-lbwvc" Jul 08 02:58:36.218551 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.218529 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj7fm\" (UniqueName: \"kubernetes.io/projected/94c47594-a484-4654-8cae-7325c0d9d581-kube-api-access-nj7fm\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:36.250718 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.250679 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lbwvc" Jul 08 02:58:36.277075 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.275353 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:36.386822 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.386650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:36.403834 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.403805 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lbwvc"] Jul 08 02:58:36.408597 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:36.408569 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9ca7be_62a3_4895_8e9c_ecaa09ea7f03.slice/crio-359cd8550f7ebd72613de8816737dde3569cbb02e71cc8532c27467663e26af7 WatchSource:0}: Error finding container 359cd8550f7ebd72613de8816737dde3569cbb02e71cc8532c27467663e26af7: Status 404 returned error can't find the container with id 359cd8550f7ebd72613de8816737dde3569cbb02e71cc8532c27467663e26af7 Jul 08 02:58:36.552543 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.552495 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:36.590849 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.590802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:36.591002 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:36.590943 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 02:58:36.591002 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:36.590959 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c4cc5dff5-lx8g9: secret "image-registry-tls" not found Jul 08 02:58:36.591107 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:36.591021 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls podName:80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:37.591003237 +0000 UTC m=+36.581451267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls") pod "image-registry-6c4cc5dff5-lx8g9" (UID: "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5") : secret "image-registry-tls" not found Jul 08 02:58:36.648679 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.648556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:36.691724 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.691689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:36.691917 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:36.691877 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:58:36.691982 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:36.691947 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls podName:94c47594-a484-4654-8cae-7325c0d9d581 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:37.691925755 +0000 UTC m=+36.682373774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls") pod "dns-default-6j82n" (UID: "94c47594-a484-4654-8cae-7325c0d9d581") : secret "dns-default-metrics-tls" not found Jul 08 02:58:36.744880 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.744843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:36.785382 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.785355 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lbwvc" event={"ID":"cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03","Type":"ContainerStarted","Data":"359cd8550f7ebd72613de8816737dde3569cbb02e71cc8532c27467663e26af7"} Jul 08 02:58:36.844989 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.844949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:36.944685 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:36.944645 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:37.522028 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:37.521171 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:58:37.522028 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:37.521176 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:58:37.522028 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:37.521317 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:58:37.525596 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:37.525563 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fsx5r\"" Jul 08 02:58:37.525596 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:37.525584 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 02:58:37.525784 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:37.525610 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 02:58:37.525784 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:37.525641 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v7mdx\"" Jul 08 02:58:37.525784 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:37.525641 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 02:58:37.525784 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:37.525581 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 02:58:37.599705 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:37.599676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:37.600129 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:37.599847 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 02:58:37.600129 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:37.599867 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c4cc5dff5-lx8g9: secret "image-registry-tls" not found Jul 08 02:58:37.600129 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:37.599929 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls podName:80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:39.599909221 +0000 UTC m=+38.590357238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls") pod "image-registry-6c4cc5dff5-lx8g9" (UID: "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5") : secret "image-registry-tls" not found Jul 08 02:58:37.700284 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:37.700247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:37.700895 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:37.700864 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:58:37.701019 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:37.700933 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls podName:94c47594-a484-4654-8cae-7325c0d9d581 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:39.700913132 +0000 UTC m=+38.691361153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls") pod "dns-default-6j82n" (UID: "94c47594-a484-4654-8cae-7325c0d9d581") : secret "dns-default-metrics-tls" not found Jul 08 02:58:39.615932 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:39.615889 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:39.616410 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:39.616082 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 02:58:39.616410 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:39.616097 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c4cc5dff5-lx8g9: secret "image-registry-tls" not found Jul 08 02:58:39.616410 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:39.616170 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls podName:80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:43.616154434 +0000 UTC m=+42.606602454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls") pod "image-registry-6c4cc5dff5-lx8g9" (UID: "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5") : secret "image-registry-tls" not found Jul 08 02:58:39.717125 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:39.717002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:39.717308 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:39.717166 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:58:39.717308 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:39.717246 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls podName:94c47594-a484-4654-8cae-7325c0d9d581 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:43.717227903 +0000 UTC m=+42.707675917 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls") pod "dns-default-6j82n" (UID: "94c47594-a484-4654-8cae-7325c0d9d581") : secret "dns-default-metrics-tls" not found Jul 08 02:58:40.797097 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:40.797062 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lbwvc" event={"ID":"cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03","Type":"ContainerStarted","Data":"eecef6d8a869988a4b63b760ba9bfde7c8f0b08bc526d9cf15586097ec4077c9"} Jul 08 02:58:40.799946 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:40.799921 2576 generic.go:356] "Generic (PLEG): container finished" podID="ab087428-a2bf-4ca5-8c12-1824860547c2" containerID="3881159e96c64196aca66fa496003a5b204e9f1b36a618a018ba5150e80125e4" exitCode=0 Jul 08 02:58:40.800078 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:40.799978 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf95s" event={"ID":"ab087428-a2bf-4ca5-8c12-1824860547c2","Type":"ContainerDied","Data":"3881159e96c64196aca66fa496003a5b204e9f1b36a618a018ba5150e80125e4"} Jul 08 02:58:40.814510 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:40.814470 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lbwvc" podStartSLOduration=2.123327546 podStartE2EDuration="5.814458718s" totalImagesPullingTime="3.691131172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:40.814001086 +0000 UTC m=+39.804449122" watchObservedRunningTime="2026-07-08 02:58:40.814458718 +0000 UTC m=+39.804906769" Jul 08 02:58:41.290580 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:41.290548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:41.343438 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:41.343402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:41.805216 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:41.805180 2576 generic.go:356] "Generic (PLEG): container finished" podID="ab087428-a2bf-4ca5-8c12-1824860547c2" containerID="a21312f3ba3d3d364b9103c604a913c80fca660fa5fc9a71dccdba3bacc9acb8" exitCode=0 Jul 08 02:58:41.805576 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:41.805266 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf95s" event={"ID":"ab087428-a2bf-4ca5-8c12-1824860547c2","Type":"ContainerDied","Data":"a21312f3ba3d3d364b9103c604a913c80fca660fa5fc9a71dccdba3bacc9acb8"} Jul 08 02:58:42.810467 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:42.810431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf95s" event={"ID":"ab087428-a2bf-4ca5-8c12-1824860547c2","Type":"ContainerStarted","Data":"276bb6d922eab39e485cde65427924abbc68880c2334265f4e390c8f4f49dbf5"} Jul 08 02:58:42.835475 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:42.835428 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nf95s" podStartSLOduration=7.297419653 podStartE2EDuration="41.83541334s" totalImagesPullingTime="34.537993687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:42.835150342 +0000 UTC m=+41.825598379" watchObservedRunningTime="2026-07-08 02:58:42.83541334 +0000 UTC m=+41.825861377" Jul 08 02:58:42.956673 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:42.956641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:43.017118 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:43.017088 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:43.645382 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:43.645346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:43.645553 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:43.645500 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 02:58:43.645553 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:43.645513 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c4cc5dff5-lx8g9: secret "image-registry-tls" not found Jul 08 02:58:43.645623 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:43.645569 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls podName:80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:51.645551427 +0000 UTC m=+50.635999444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls") pod "image-registry-6c4cc5dff5-lx8g9" (UID: "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5") : secret "image-registry-tls" not found Jul 08 02:58:43.746612 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:43.746570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:43.746763 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:43.746730 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:58:43.746810 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:43.746799 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls podName:94c47594-a484-4654-8cae-7325c0d9d581 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:51.74678467 +0000 UTC m=+50.737232689 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls") pod "dns-default-6j82n" (UID: "94c47594-a484-4654-8cae-7325c0d9d581") : secret "dns-default-metrics-tls" not found Jul 08 02:58:45.814795 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:45.814736 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:45.940301 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:45.940260 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:46.017891 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:46.017846 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:46.119607 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:46.119516 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:46.228449 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:46.228414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:47.413431 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:47.413397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:49.239826 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.239791 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ddgb2"] Jul 08 02:58:49.282416 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.282384 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx"] Jul 08 02:58:49.282557 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.282500 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ddgb2" Jul 08 02:58:49.285358 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.285338 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-sdlql\"" Jul 08 02:58:49.295441 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.295424 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cfrk8"] Jul 08 02:58:49.295556 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.295540 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" Jul 08 02:58:49.298414 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.298394 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 08 02:58:49.298526 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.298433 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 08 02:58:49.298526 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.298434 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vrpbl\"" Jul 08 02:58:49.298526 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.298478 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 08 02:58:49.298526 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.298486 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 08 02:58:49.311882 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.311864 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-jz8tg"] Jul 08 02:58:49.311981 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.311967 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cfrk8" Jul 08 02:58:49.314722 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.314706 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 08 02:58:49.314918 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.314902 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 08 02:58:49.314993 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.314944 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jzjzl\"" Jul 08 02:58:49.338101 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.338075 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-lrzh5"] Jul 08 02:58:49.338225 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.338181 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.341280 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.341260 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 08 02:58:49.341396 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.341379 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lgktb\"" Jul 08 02:58:49.341467 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.341397 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 02:58:49.341467 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.341397 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 08 02:58:49.341575 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.341399 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 02:58:49.348523 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.348492 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 08 02:58:49.358092 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.358068 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ddgb2"] Jul 08 02:58:49.358192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.358104 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cfrk8"] Jul 08 02:58:49.358192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.358110 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:49.358192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.358127 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx"] Jul 08 02:58:49.358192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.358148 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-lrzh5"] Jul 08 02:58:49.358192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.358166 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-jz8tg"] Jul 08 02:58:49.358447 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.358200 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5c46bc767-r4rqj"] Jul 08 02:58:49.361087 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.361064 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 08 02:58:49.361087 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.361076 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 08 02:58:49.361222 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.361071 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 08 02:58:49.361222 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.361073 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 08 02:58:49.361222 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.361095 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ssh5r\"" Jul 08 02:58:49.366631 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.366607 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 08 02:58:49.378248 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.378228 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c46bc767-r4rqj"] Jul 08 02:58:49.378336 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.378300 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:49.380879 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.380861 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 08 02:58:49.380986 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.380886 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 08 02:58:49.380986 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.380956 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mtkbk\"" Jul 08 02:58:49.380986 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.380955 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 02:58:49.381127 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.381002 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 08 02:58:49.381127 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.381002 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 02:58:49.381198 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.381184 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 08 02:58:49.391713 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.391692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82dlp\" (UniqueName: \"kubernetes.io/projected/685a7232-5ff7-4f67-b949-1c41b98bb3c3-kube-api-access-82dlp\") pod \"kube-storage-version-migrator-operator-846bf668f7-45htx\" (UID: \"685a7232-5ff7-4f67-b949-1c41b98bb3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" Jul 08 02:58:49.391835 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.391811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdq5l\" (UniqueName: \"kubernetes.io/projected/d43efad6-8e7d-4813-a798-c0bed80895e3-kube-api-access-rdq5l\") pod \"network-check-source-989f8b8dc-ddgb2\" (UID: \"d43efad6-8e7d-4813-a798-c0bed80895e3\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ddgb2" Jul 08 02:58:49.391883 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.391856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/685a7232-5ff7-4f67-b949-1c41b98bb3c3-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-45htx\" (UID: \"685a7232-5ff7-4f67-b949-1c41b98bb3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" Jul 08 02:58:49.391919 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.391881 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/685a7232-5ff7-4f67-b949-1c41b98bb3c3-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-45htx\" (UID: \"685a7232-5ff7-4f67-b949-1c41b98bb3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" Jul 08 02:58:49.442751 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.442726 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z"] Jul 08 02:58:49.468069 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.468020 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5"] Jul 08 02:58:49.468202 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.468142 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" Jul 08 02:58:49.470776 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.470751 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 08 02:58:49.470903 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.470837 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 08 02:58:49.470961 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.470904 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 08 02:58:49.470961 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.470917 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tvjqr\"" Jul 08 02:58:49.492169 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492095 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk"] Jul 08 02:58:49.492308 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492203 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:58:49.492396 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdq5l\" (UniqueName: \"kubernetes.io/projected/d43efad6-8e7d-4813-a798-c0bed80895e3-kube-api-access-rdq5l\") pod \"network-check-source-989f8b8dc-ddgb2\" (UID: \"d43efad6-8e7d-4813-a798-c0bed80895e3\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ddgb2" Jul 08 02:58:49.492463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f73570e2-30eb-4cb3-9d67-0b6689c9f312-tmp\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.492521 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492452 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f73570e2-30eb-4cb3-9d67-0b6689c9f312-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.492521 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492507 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4220e41-4e35-4cc8-a67b-db04dd99e5cb-config\") pod \"console-operator-68b757fb67-lrzh5\" (UID: \"d4220e41-4e35-4cc8-a67b-db04dd99e5cb\") " pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:49.492625 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492534 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:49.492625 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-stats-auth\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:49.492724 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/685a7232-5ff7-4f67-b949-1c41b98bb3c3-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-45htx\" (UID: \"685a7232-5ff7-4f67-b949-1c41b98bb3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" Jul 08 02:58:49.492724 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/685a7232-5ff7-4f67-b949-1c41b98bb3c3-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-45htx\" (UID: \"685a7232-5ff7-4f67-b949-1c41b98bb3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" Jul 08 02:58:49.492724 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492712 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f73570e2-30eb-4cb3-9d67-0b6689c9f312-snapshots\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.492867 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82dlp\" (UniqueName: \"kubernetes.io/projected/685a7232-5ff7-4f67-b949-1c41b98bb3c3-kube-api-access-82dlp\") pod \"kube-storage-version-migrator-operator-846bf668f7-45htx\" (UID: \"685a7232-5ff7-4f67-b949-1c41b98bb3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" Jul 08 02:58:49.492867 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snjj4\" (UniqueName: \"kubernetes.io/projected/3a708c11-6608-4fd6-8f47-f2d9ded1569a-kube-api-access-snjj4\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:49.492867 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f73570e2-30eb-4cb3-9d67-0b6689c9f312-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.492867 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4220e41-4e35-4cc8-a67b-db04dd99e5cb-serving-cert\") pod \"console-operator-68b757fb67-lrzh5\" (UID: \"d4220e41-4e35-4cc8-a67b-db04dd99e5cb\") " pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:49.493078 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492881 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkpm\" (UniqueName: \"kubernetes.io/projected/d4220e41-4e35-4cc8-a67b-db04dd99e5cb-kube-api-access-nzkpm\") pod \"console-operator-68b757fb67-lrzh5\" (UID: \"d4220e41-4e35-4cc8-a67b-db04dd99e5cb\") " pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:49.493078 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzm5g\" (UniqueName: \"kubernetes.io/projected/f73570e2-30eb-4cb3-9d67-0b6689c9f312-kube-api-access-nzm5g\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.493078 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.492984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbj5\" (UniqueName: \"kubernetes.io/projected/df234fb8-8adb-4814-836d-7dfb0ea5a135-kube-api-access-mhbj5\") pod \"volume-data-source-validator-5d456fdbcd-cfrk8\" (UID: \"df234fb8-8adb-4814-836d-7dfb0ea5a135\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cfrk8" Jul 08 02:58:49.493078 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.493011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4220e41-4e35-4cc8-a67b-db04dd99e5cb-trusted-ca\") pod \"console-operator-68b757fb67-lrzh5\" (UID: \"d4220e41-4e35-4cc8-a67b-db04dd99e5cb\") " pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:49.493256 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.493073 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f73570e2-30eb-4cb3-9d67-0b6689c9f312-serving-cert\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.493256 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.493127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-default-certificate\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:49.493256 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.493143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:49.493345 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.493305 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/685a7232-5ff7-4f67-b949-1c41b98bb3c3-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-45htx\" (UID: \"685a7232-5ff7-4f67-b949-1c41b98bb3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" Jul 08 02:58:49.495274 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.495254 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 08 02:58:49.495431 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.495416 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 02:58:49.495725 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.495703 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-h2vn8\"" Jul 08 02:58:49.495919 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.495818 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 08 02:58:49.495919 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.495906 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 02:58:49.496994 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.496974 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/685a7232-5ff7-4f67-b949-1c41b98bb3c3-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-45htx\" (UID: \"685a7232-5ff7-4f67-b949-1c41b98bb3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" Jul 08 02:58:49.501636 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.501611 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82dlp\" (UniqueName: \"kubernetes.io/projected/685a7232-5ff7-4f67-b949-1c41b98bb3c3-kube-api-access-82dlp\") pod \"kube-storage-version-migrator-operator-846bf668f7-45htx\" (UID: \"685a7232-5ff7-4f67-b949-1c41b98bb3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" Jul 08 02:58:49.501891 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.501873 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdq5l\" (UniqueName: \"kubernetes.io/projected/d43efad6-8e7d-4813-a798-c0bed80895e3-kube-api-access-rdq5l\") pod \"network-check-source-989f8b8dc-ddgb2\" (UID: \"d43efad6-8e7d-4813-a798-c0bed80895e3\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ddgb2" Jul 08 02:58:49.504916 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.504895 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:49.516306 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.516285 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z"] Jul 08 02:58:49.516403 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.516325 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk"] Jul 08 02:58:49.516403 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.516350 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5"] Jul 08 02:58:49.516403 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.516394 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" Jul 08 02:58:49.518952 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.518933 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 08 02:58:49.519034 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.518996 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 08 02:58:49.519249 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.519225 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-q26bc\"" Jul 08 02:58:49.519249 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.519230 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 08 02:58:49.519398 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.519280 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 08 02:58:49.561479 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.561455 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:49.591389 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.591367 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ddgb2" Jul 08 02:58:49.593676 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.593657 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f73570e2-30eb-4cb3-9d67-0b6689c9f312-tmp\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.593727 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.593688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f73570e2-30eb-4cb3-9d67-0b6689c9f312-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.593727 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.593706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4220e41-4e35-4cc8-a67b-db04dd99e5cb-config\") pod \"console-operator-68b757fb67-lrzh5\" (UID: \"d4220e41-4e35-4cc8-a67b-db04dd99e5cb\") " pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:49.593727 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.593722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:49.593827 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.593738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-stats-auth\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:49.593827 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.593764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-4bs9z\" (UID: \"47662f67-2e4a-40ef-8cf9-a55730c0b6c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" Jul 08 02:58:49.593916 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.593825 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:58:49.593916 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:49.593841 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 02:58:49.593916 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.593850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k89n\" (UniqueName: \"kubernetes.io/projected/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-kube-api-access-5k89n\") pod \"cluster-samples-operator-56fcf86bf-4bs9z\" (UID: \"47662f67-2e4a-40ef-8cf9-a55730c0b6c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" Jul 08 02:58:49.593916 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.593889 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f73570e2-30eb-4cb3-9d67-0b6689c9f312-snapshots\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.593916 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:49.593903 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs podName:3a708c11-6608-4fd6-8f47-f2d9ded1569a nodeName:}" failed. No retries permitted until 2026-07-08 02:58:50.09388556 +0000 UTC m=+49.084333583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs") pod "router-default-5c46bc767-r4rqj" (UID: "3a708c11-6608-4fd6-8f47-f2d9ded1569a") : secret "router-metrics-certs-default" not found Jul 08 02:58:49.594180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.593936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snjj4\" (UniqueName: \"kubernetes.io/projected/3a708c11-6608-4fd6-8f47-f2d9ded1569a-kube-api-access-snjj4\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:49.594180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.593973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f73570e2-30eb-4cb3-9d67-0b6689c9f312-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.594180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.593994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4220e41-4e35-4cc8-a67b-db04dd99e5cb-serving-cert\") pod \"console-operator-68b757fb67-lrzh5\" (UID: \"d4220e41-4e35-4cc8-a67b-db04dd99e5cb\") " pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:49.594180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.594019 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:58:49.594180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.594071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkpm\" (UniqueName: \"kubernetes.io/projected/d4220e41-4e35-4cc8-a67b-db04dd99e5cb-kube-api-access-nzkpm\") pod \"console-operator-68b757fb67-lrzh5\" (UID: \"d4220e41-4e35-4cc8-a67b-db04dd99e5cb\") " pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:49.594180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.594106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzm5g\" (UniqueName: \"kubernetes.io/projected/f73570e2-30eb-4cb3-9d67-0b6689c9f312-kube-api-access-nzm5g\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.594180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.594128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbj5\" (UniqueName: \"kubernetes.io/projected/df234fb8-8adb-4814-836d-7dfb0ea5a135-kube-api-access-mhbj5\") pod \"volume-data-source-validator-5d456fdbcd-cfrk8\" (UID: \"df234fb8-8adb-4814-836d-7dfb0ea5a135\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cfrk8" Jul 08 02:58:49.594180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.594157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkw46\" (UniqueName: \"kubernetes.io/projected/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-kube-api-access-nkw46\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:58:49.594575 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.594186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4220e41-4e35-4cc8-a67b-db04dd99e5cb-trusted-ca\") pod \"console-operator-68b757fb67-lrzh5\" (UID: \"d4220e41-4e35-4cc8-a67b-db04dd99e5cb\") " pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:49.594575 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.594196 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f73570e2-30eb-4cb3-9d67-0b6689c9f312-tmp\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.594575 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.594215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f73570e2-30eb-4cb3-9d67-0b6689c9f312-serving-cert\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.594575 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.594279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-default-certificate\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:49.594575 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.594312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:49.594575 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.594334 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f73570e2-30eb-4cb3-9d67-0b6689c9f312-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.594575 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.594429 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4220e41-4e35-4cc8-a67b-db04dd99e5cb-config\") pod \"console-operator-68b757fb67-lrzh5\" (UID: \"d4220e41-4e35-4cc8-a67b-db04dd99e5cb\") " pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:49.594575 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.594521 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f73570e2-30eb-4cb3-9d67-0b6689c9f312-snapshots\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.594955 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:49.594746 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle podName:3a708c11-6608-4fd6-8f47-f2d9ded1569a nodeName:}" failed. No retries permitted until 2026-07-08 02:58:50.094728427 +0000 UTC m=+49.085176464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle") pod "router-default-5c46bc767-r4rqj" (UID: "3a708c11-6608-4fd6-8f47-f2d9ded1569a") : configmap references non-existent config key: service-ca.crt Jul 08 02:58:49.595161 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.595138 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f73570e2-30eb-4cb3-9d67-0b6689c9f312-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.595293 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.595272 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4220e41-4e35-4cc8-a67b-db04dd99e5cb-trusted-ca\") pod \"console-operator-68b757fb67-lrzh5\" (UID: \"d4220e41-4e35-4cc8-a67b-db04dd99e5cb\") " pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:49.596779 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.596755 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-stats-auth\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:49.597023 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.597002 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f73570e2-30eb-4cb3-9d67-0b6689c9f312-serving-cert\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.597117 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.597099 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4220e41-4e35-4cc8-a67b-db04dd99e5cb-serving-cert\") pod \"console-operator-68b757fb67-lrzh5\" (UID: \"d4220e41-4e35-4cc8-a67b-db04dd99e5cb\") " pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:49.597586 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.597560 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-default-certificate\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:49.603445 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.603422 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" Jul 08 02:58:49.604031 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.604004 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkpm\" (UniqueName: \"kubernetes.io/projected/d4220e41-4e35-4cc8-a67b-db04dd99e5cb-kube-api-access-nzkpm\") pod \"console-operator-68b757fb67-lrzh5\" (UID: \"d4220e41-4e35-4cc8-a67b-db04dd99e5cb\") " pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:49.604144 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.604067 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snjj4\" (UniqueName: \"kubernetes.io/projected/3a708c11-6608-4fd6-8f47-f2d9ded1569a-kube-api-access-snjj4\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:49.604729 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.604710 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzm5g\" (UniqueName: \"kubernetes.io/projected/f73570e2-30eb-4cb3-9d67-0b6689c9f312-kube-api-access-nzm5g\") pod \"insights-operator-5c847dcfc4-jz8tg\" (UID: \"f73570e2-30eb-4cb3-9d67-0b6689c9f312\") " pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.604830 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.604809 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbj5\" (UniqueName: \"kubernetes.io/projected/df234fb8-8adb-4814-836d-7dfb0ea5a135-kube-api-access-mhbj5\") pod \"volume-data-source-validator-5d456fdbcd-cfrk8\" (UID: \"df234fb8-8adb-4814-836d-7dfb0ea5a135\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cfrk8" Jul 08 02:58:49.620112 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.620094 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cfrk8" Jul 08 02:58:49.647674 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.647644 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" Jul 08 02:58:49.667973 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.667860 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:49.698359 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.696019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-4bs9z\" (UID: \"47662f67-2e4a-40ef-8cf9-a55730c0b6c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" Jul 08 02:58:49.698359 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.696099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:58:49.698359 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.696127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k89n\" (UniqueName: \"kubernetes.io/projected/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-kube-api-access-5k89n\") pod \"cluster-samples-operator-56fcf86bf-4bs9z\" (UID: \"47662f67-2e4a-40ef-8cf9-a55730c0b6c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" Jul 08 02:58:49.698359 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.696161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/632607e8-3768-4d7c-8c3c-7bdcc00ad8d6-config\") pod \"service-ca-operator-5d46d9ffd4-5dkrk\" (UID: \"632607e8-3768-4d7c-8c3c-7bdcc00ad8d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" Jul 08 02:58:49.698359 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.696217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:58:49.698359 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.696268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkw46\" (UniqueName: \"kubernetes.io/projected/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-kube-api-access-nkw46\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:58:49.698359 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.696297 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rhgd\" (UniqueName: \"kubernetes.io/projected/632607e8-3768-4d7c-8c3c-7bdcc00ad8d6-kube-api-access-2rhgd\") pod \"service-ca-operator-5d46d9ffd4-5dkrk\" (UID: \"632607e8-3768-4d7c-8c3c-7bdcc00ad8d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" Jul 08 02:58:49.698359 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.696333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/632607e8-3768-4d7c-8c3c-7bdcc00ad8d6-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-5dkrk\" (UID: \"632607e8-3768-4d7c-8c3c-7bdcc00ad8d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" Jul 08 02:58:49.698359 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:49.696532 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 02:58:49.698359 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:49.696602 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls podName:47662f67-2e4a-40ef-8cf9-a55730c0b6c6 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:50.196583341 +0000 UTC m=+49.187031369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-4bs9z" (UID: "47662f67-2e4a-40ef-8cf9-a55730c0b6c6") : secret "samples-operator-tls" not found Jul 08 02:58:49.698359 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.697558 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:58:49.698359 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:49.697886 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 02:58:49.698359 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:49.697933 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls podName:cce8a6b5-0ecb-4ce6-bad9-e4387039a932 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:50.197918098 +0000 UTC m=+49.188366128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-pqdv5" (UID: "cce8a6b5-0ecb-4ce6-bad9-e4387039a932") : secret "cluster-monitoring-operator-tls" not found Jul 08 02:58:49.718168 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.717639 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkw46\" (UniqueName: \"kubernetes.io/projected/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-kube-api-access-nkw46\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:58:49.720003 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.719976 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k89n\" (UniqueName: \"kubernetes.io/projected/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-kube-api-access-5k89n\") pod \"cluster-samples-operator-56fcf86bf-4bs9z\" (UID: \"47662f67-2e4a-40ef-8cf9-a55730c0b6c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" Jul 08 02:58:49.775803 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.774658 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ddgb2"] Jul 08 02:58:49.783966 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:49.783871 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43efad6_8e7d_4813_a798_c0bed80895e3.slice/crio-aaea747956e50106039f04dd328863fef056a3ea99cf53c2bd3e8a6383ad7eec WatchSource:0}: Error finding container aaea747956e50106039f04dd328863fef056a3ea99cf53c2bd3e8a6383ad7eec: Status 404 returned error can't find the container with id aaea747956e50106039f04dd328863fef056a3ea99cf53c2bd3e8a6383ad7eec Jul 08 02:58:49.796921 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.796871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rhgd\" (UniqueName: \"kubernetes.io/projected/632607e8-3768-4d7c-8c3c-7bdcc00ad8d6-kube-api-access-2rhgd\") pod \"service-ca-operator-5d46d9ffd4-5dkrk\" (UID: \"632607e8-3768-4d7c-8c3c-7bdcc00ad8d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" Jul 08 02:58:49.796921 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.796916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/632607e8-3768-4d7c-8c3c-7bdcc00ad8d6-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-5dkrk\" (UID: \"632607e8-3768-4d7c-8c3c-7bdcc00ad8d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" Jul 08 02:58:49.797090 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.797071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/632607e8-3768-4d7c-8c3c-7bdcc00ad8d6-config\") pod \"service-ca-operator-5d46d9ffd4-5dkrk\" (UID: \"632607e8-3768-4d7c-8c3c-7bdcc00ad8d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" Jul 08 02:58:49.797831 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.797777 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/632607e8-3768-4d7c-8c3c-7bdcc00ad8d6-config\") pod \"service-ca-operator-5d46d9ffd4-5dkrk\" (UID: \"632607e8-3768-4d7c-8c3c-7bdcc00ad8d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" Jul 08 02:58:49.800603 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.800557 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/632607e8-3768-4d7c-8c3c-7bdcc00ad8d6-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-5dkrk\" (UID: \"632607e8-3768-4d7c-8c3c-7bdcc00ad8d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" Jul 08 02:58:49.809995 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.809973 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rhgd\" (UniqueName: \"kubernetes.io/projected/632607e8-3768-4d7c-8c3c-7bdcc00ad8d6-kube-api-access-2rhgd\") pod \"service-ca-operator-5d46d9ffd4-5dkrk\" (UID: \"632607e8-3768-4d7c-8c3c-7bdcc00ad8d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" Jul 08 02:58:49.824241 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.824175 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" Jul 08 02:58:49.830449 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.829876 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ddgb2" event={"ID":"d43efad6-8e7d-4813-a798-c0bed80895e3","Type":"ContainerStarted","Data":"aaea747956e50106039f04dd328863fef056a3ea99cf53c2bd3e8a6383ad7eec"} Jul 08 02:58:49.831989 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.831935 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx"] Jul 08 02:58:49.872009 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.871979 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cfrk8"] Jul 08 02:58:49.875760 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:49.875720 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf234fb8_8adb_4814_836d_7dfb0ea5a135.slice/crio-72b720fb964cfa82d69b934695456d19f7ca34fd63aa2c138bfd97f854932c21 WatchSource:0}: Error finding container 72b720fb964cfa82d69b934695456d19f7ca34fd63aa2c138bfd97f854932c21: Status 404 returned error can't find the container with id 72b720fb964cfa82d69b934695456d19f7ca34fd63aa2c138bfd97f854932c21 Jul 08 02:58:49.912263 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.910244 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-jz8tg"] Jul 08 02:58:49.913237 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.913202 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-lrzh5"] Jul 08 02:58:49.913441 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:49.913393 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73570e2_30eb_4cb3_9d67_0b6689c9f312.slice/crio-b0171390bf65ca603ceb49eb280b08d9f7c63746a158a7cde1bce63f14cc8f18 WatchSource:0}: Error finding container b0171390bf65ca603ceb49eb280b08d9f7c63746a158a7cde1bce63f14cc8f18: Status 404 returned error can't find the container with id b0171390bf65ca603ceb49eb280b08d9f7c63746a158a7cde1bce63f14cc8f18 Jul 08 02:58:49.918226 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:49.918192 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4220e41_4e35_4cc8_a67b_db04dd99e5cb.slice/crio-1a5e53042fefd1969b8da20f5428cb9d29b102d4fe29b03b96a47fb1c7ab993f WatchSource:0}: Error finding container 1a5e53042fefd1969b8da20f5428cb9d29b102d4fe29b03b96a47fb1c7ab993f: Status 404 returned error can't find the container with id 1a5e53042fefd1969b8da20f5428cb9d29b102d4fe29b03b96a47fb1c7ab993f Jul 08 02:58:49.968169 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.968139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:49.989392 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:49.989366 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk"] Jul 08 02:58:49.993177 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:49.993152 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632607e8_3768_4d7c_8c3c_7bdcc00ad8d6.slice/crio-8ad7d438fee6abed9808c4e1caf87c638b8a7429ff16f30ba3475622520b2541 WatchSource:0}: Error finding container 8ad7d438fee6abed9808c4e1caf87c638b8a7429ff16f30ba3475622520b2541: Status 404 returned error can't find the container with id 8ad7d438fee6abed9808c4e1caf87c638b8a7429ff16f30ba3475622520b2541 Jul 08 02:58:50.068132 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:50.068030 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:50.099283 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:50.099250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:50.099423 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:50.099322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:50.099503 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:50.099421 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle podName:3a708c11-6608-4fd6-8f47-f2d9ded1569a nodeName:}" failed. No retries permitted until 2026-07-08 02:58:51.099404426 +0000 UTC m=+50.089852445 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle") pod "router-default-5c46bc767-r4rqj" (UID: "3a708c11-6608-4fd6-8f47-f2d9ded1569a") : configmap references non-existent config key: service-ca.crt Jul 08 02:58:50.099503 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:50.099436 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 02:58:50.099503 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:50.099491 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs podName:3a708c11-6608-4fd6-8f47-f2d9ded1569a nodeName:}" failed. No retries permitted until 2026-07-08 02:58:51.099473736 +0000 UTC m=+50.089921760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs") pod "router-default-5c46bc767-r4rqj" (UID: "3a708c11-6608-4fd6-8f47-f2d9ded1569a") : secret "router-metrics-certs-default" not found Jul 08 02:58:50.199977 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:50.199947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-4bs9z\" (UID: \"47662f67-2e4a-40ef-8cf9-a55730c0b6c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" Jul 08 02:58:50.200139 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:50.200007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:58:50.200139 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:50.200120 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 02:58:50.200139 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:50.200119 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 02:58:50.200233 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:50.200173 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls podName:cce8a6b5-0ecb-4ce6-bad9-e4387039a932 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:51.200160475 +0000 UTC m=+50.190608490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-pqdv5" (UID: "cce8a6b5-0ecb-4ce6-bad9-e4387039a932") : secret "cluster-monitoring-operator-tls" not found Jul 08 02:58:50.200233 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:50.200187 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls podName:47662f67-2e4a-40ef-8cf9-a55730c0b6c6 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:51.20018015 +0000 UTC m=+50.190628166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-4bs9z" (UID: "47662f67-2e4a-40ef-8cf9-a55730c0b6c6") : secret "samples-operator-tls" not found Jul 08 02:58:50.836267 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:50.836177 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cfrk8" event={"ID":"df234fb8-8adb-4814-836d-7dfb0ea5a135","Type":"ContainerStarted","Data":"72b720fb964cfa82d69b934695456d19f7ca34fd63aa2c138bfd97f854932c21"} Jul 08 02:58:50.840272 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:50.840198 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" event={"ID":"d4220e41-4e35-4cc8-a67b-db04dd99e5cb","Type":"ContainerStarted","Data":"1a5e53042fefd1969b8da20f5428cb9d29b102d4fe29b03b96a47fb1c7ab993f"} Jul 08 02:58:50.841787 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:50.841716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" event={"ID":"632607e8-3768-4d7c-8c3c-7bdcc00ad8d6","Type":"ContainerStarted","Data":"8ad7d438fee6abed9808c4e1caf87c638b8a7429ff16f30ba3475622520b2541"} Jul 08 02:58:50.855826 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:50.855716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" event={"ID":"f73570e2-30eb-4cb3-9d67-0b6689c9f312","Type":"ContainerStarted","Data":"b0171390bf65ca603ceb49eb280b08d9f7c63746a158a7cde1bce63f14cc8f18"} Jul 08 02:58:50.858683 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:50.858653 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" event={"ID":"685a7232-5ff7-4f67-b949-1c41b98bb3c3","Type":"ContainerStarted","Data":"d8fa8bad50671036558a617cdf76a7680f7ebf224ecfe9a64e5cc25b2f3ea533"} Jul 08 02:58:51.108899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:51.107760 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:51.108899 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:51.107858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:51.108899 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:51.108092 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 02:58:51.108899 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:51.108155 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs podName:3a708c11-6608-4fd6-8f47-f2d9ded1569a nodeName:}" failed. No retries permitted until 2026-07-08 02:58:53.108138013 +0000 UTC m=+52.098586044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs") pod "router-default-5c46bc767-r4rqj" (UID: "3a708c11-6608-4fd6-8f47-f2d9ded1569a") : secret "router-metrics-certs-default" not found Jul 08 02:58:51.108899 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:51.108679 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle podName:3a708c11-6608-4fd6-8f47-f2d9ded1569a nodeName:}" failed. No retries permitted until 2026-07-08 02:58:53.108662035 +0000 UTC m=+52.099110066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle") pod "router-default-5c46bc767-r4rqj" (UID: "3a708c11-6608-4fd6-8f47-f2d9ded1569a") : configmap references non-existent config key: service-ca.crt Jul 08 02:58:51.209542 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:51.208571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:58:51.209542 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:51.208749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-4bs9z\" (UID: \"47662f67-2e4a-40ef-8cf9-a55730c0b6c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" Jul 08 02:58:51.209542 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:51.208901 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 02:58:51.209542 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:51.208998 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls podName:47662f67-2e4a-40ef-8cf9-a55730c0b6c6 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:53.208977823 +0000 UTC m=+52.199425858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-4bs9z" (UID: "47662f67-2e4a-40ef-8cf9-a55730c0b6c6") : secret "samples-operator-tls" not found Jul 08 02:58:51.209542 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:51.209456 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 02:58:51.209542 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:51.209504 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls podName:cce8a6b5-0ecb-4ce6-bad9-e4387039a932 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:53.209487662 +0000 UTC m=+52.199935677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-pqdv5" (UID: "cce8a6b5-0ecb-4ce6-bad9-e4387039a932") : secret "cluster-monitoring-operator-tls" not found Jul 08 02:58:51.714934 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:51.714885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:58:51.715131 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:51.715114 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 02:58:51.715131 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:51.715131 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c4cc5dff5-lx8g9: secret "image-registry-tls" not found Jul 08 02:58:51.715244 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:51.715189 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls podName:80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:07.715170706 +0000 UTC m=+66.705618723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls") pod "image-registry-6c4cc5dff5-lx8g9" (UID: "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5") : secret "image-registry-tls" not found Jul 08 02:58:51.816333 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:51.816288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:58:51.816550 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:51.816530 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 02:58:51.816673 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:51.816634 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls podName:94c47594-a484-4654-8cae-7325c0d9d581 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:07.816615942 +0000 UTC m=+66.807063962 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls") pod "dns-default-6j82n" (UID: "94c47594-a484-4654-8cae-7325c0d9d581") : secret "dns-default-metrics-tls" not found Jul 08 02:58:53.129183 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:53.129099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:53.129676 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:53.129209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:53.129676 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:53.129287 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle podName:3a708c11-6608-4fd6-8f47-f2d9ded1569a nodeName:}" failed. No retries permitted until 2026-07-08 02:58:57.129265541 +0000 UTC m=+56.119713561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle") pod "router-default-5c46bc767-r4rqj" (UID: "3a708c11-6608-4fd6-8f47-f2d9ded1569a") : configmap references non-existent config key: service-ca.crt Jul 08 02:58:53.129676 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:53.129336 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 02:58:53.129676 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:53.129396 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs podName:3a708c11-6608-4fd6-8f47-f2d9ded1569a nodeName:}" failed. No retries permitted until 2026-07-08 02:58:57.129379622 +0000 UTC m=+56.119827657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs") pod "router-default-5c46bc767-r4rqj" (UID: "3a708c11-6608-4fd6-8f47-f2d9ded1569a") : secret "router-metrics-certs-default" not found Jul 08 02:58:53.230231 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:53.230171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-4bs9z\" (UID: \"47662f67-2e4a-40ef-8cf9-a55730c0b6c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" Jul 08 02:58:53.230408 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:53.230270 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:58:53.230408 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:53.230323 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 02:58:53.230491 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:53.230408 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls podName:47662f67-2e4a-40ef-8cf9-a55730c0b6c6 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:57.230386293 +0000 UTC m=+56.220834311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-4bs9z" (UID: "47662f67-2e4a-40ef-8cf9-a55730c0b6c6") : secret "samples-operator-tls" not found Jul 08 02:58:53.230491 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:53.230442 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 02:58:53.230491 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:53.230486 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls podName:cce8a6b5-0ecb-4ce6-bad9-e4387039a932 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:57.230474271 +0000 UTC m=+56.220922299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-pqdv5" (UID: "cce8a6b5-0ecb-4ce6-bad9-e4387039a932") : secret "cluster-monitoring-operator-tls" not found Jul 08 02:58:55.607729 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:55.607690 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw"] Jul 08 02:58:55.641001 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:55.640967 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw"] Jul 08 02:58:55.641190 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:55.641134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" Jul 08 02:58:55.643900 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:55.643873 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 08 02:58:55.644038 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:55.643928 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ck74d\"" Jul 08 02:58:55.644038 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:55.643953 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 08 02:58:55.649592 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:55.649564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rrkzw\" (UID: \"5a53b345-6e70-44ee-b799-4f4c24c1d094\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" Jul 08 02:58:55.649728 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:55.649708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5a53b345-6e70-44ee-b799-4f4c24c1d094-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-rrkzw\" (UID: \"5a53b345-6e70-44ee-b799-4f4c24c1d094\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" Jul 08 02:58:55.729981 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:55.729946 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:55.751101 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:55.751062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rrkzw\" (UID: \"5a53b345-6e70-44ee-b799-4f4c24c1d094\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" Jul 08 02:58:55.751280 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:55.751166 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 02:58:55.751280 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:55.751221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5a53b345-6e70-44ee-b799-4f4c24c1d094-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-rrkzw\" (UID: \"5a53b345-6e70-44ee-b799-4f4c24c1d094\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" Jul 08 02:58:55.751365 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:55.751316 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert podName:5a53b345-6e70-44ee-b799-4f4c24c1d094 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:56.251221734 +0000 UTC m=+55.241669753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-rrkzw" (UID: "5a53b345-6e70-44ee-b799-4f4c24c1d094") : secret "networking-console-plugin-cert" not found Jul 08 02:58:55.751879 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:55.751860 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5a53b345-6e70-44ee-b799-4f4c24c1d094-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-rrkzw\" (UID: \"5a53b345-6e70-44ee-b799-4f4c24c1d094\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" Jul 08 02:58:55.828464 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:55.828429 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:55.928472 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:55.928434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:56.028944 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.028834 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:56.256341 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.255821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rrkzw\" (UID: \"5a53b345-6e70-44ee-b799-4f4c24c1d094\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" Jul 08 02:58:56.256341 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:56.255986 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 02:58:56.256341 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:56.256035 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert podName:5a53b345-6e70-44ee-b799-4f4c24c1d094 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:57.256020132 +0000 UTC m=+56.246468146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-rrkzw" (UID: "5a53b345-6e70-44ee-b799-4f4c24c1d094") : secret "networking-console-plugin-cert" not found Jul 08 02:58:56.883868 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.883831 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/0.log" Jul 08 02:58:56.884357 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.883881 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4220e41-4e35-4cc8-a67b-db04dd99e5cb" containerID="2b0b02ef5c98ee2417d05a0001b2558b04a1ff1e797ffb4b961786184e97632a" exitCode=255 Jul 08 02:58:56.884357 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.884263 2576 scope.go:117] "RemoveContainer" containerID="2b0b02ef5c98ee2417d05a0001b2558b04a1ff1e797ffb4b961786184e97632a" Jul 08 02:58:56.884590 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.884569 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" event={"ID":"d4220e41-4e35-4cc8-a67b-db04dd99e5cb","Type":"ContainerDied","Data":"2b0b02ef5c98ee2417d05a0001b2558b04a1ff1e797ffb4b961786184e97632a"} Jul 08 02:58:56.886237 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.886211 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" event={"ID":"632607e8-3768-4d7c-8c3c-7bdcc00ad8d6","Type":"ContainerStarted","Data":"24a04ab81c83cd7df7c794b9c10482da206126c611b2a2ccbcc78feb38ccb67f"} Jul 08 02:58:56.888213 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.888157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ddgb2" event={"ID":"d43efad6-8e7d-4813-a798-c0bed80895e3","Type":"ContainerStarted","Data":"fdfe69c86e61c9ae20e60943bb7ea16ee8e4cc4690caad2c5f4bd3ff4af76c62"} Jul 08 02:58:56.891080 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.889890 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" event={"ID":"f73570e2-30eb-4cb3-9d67-0b6689c9f312","Type":"ContainerStarted","Data":"d4a6139818a175c772071d9c7f217949f23d8b30e3a12582a6281ce5e0091f36"} Jul 08 02:58:56.892524 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.892503 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" event={"ID":"685a7232-5ff7-4f67-b949-1c41b98bb3c3","Type":"ContainerStarted","Data":"28d45af4ea3605a70cb6732d686594ee43e2c6c5c0c7c1e09e603b98c94bbbc1"} Jul 08 02:58:56.893859 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.893838 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cfrk8" event={"ID":"df234fb8-8adb-4814-836d-7dfb0ea5a135","Type":"ContainerStarted","Data":"87964c2823e6f75342a801223ed340a3bff4ecbbe81ea02dbe44f62e98df86a7"} Jul 08 02:58:56.918467 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.918418 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ddgb2" podStartSLOduration=1.668827613 podStartE2EDuration="7.918402738s" totalImagesPullingTime="6.249575125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:56.91802659 +0000 UTC m=+55.908474627" watchObservedRunningTime="2026-07-08 02:58:56.918402738 +0000 UTC m=+55.908850754" Jul 08 02:58:56.934735 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.934601 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" podStartSLOduration=1.7515432629999999 podStartE2EDuration="7.934579655s" totalImagesPullingTime="6.183036392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:56.934308241 +0000 UTC m=+55.924756279" watchObservedRunningTime="2026-07-08 02:58:56.934579655 +0000 UTC m=+55.925027693" Jul 08 02:58:56.953180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.951747 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cfrk8" podStartSLOduration=1.8160859710000001 podStartE2EDuration="7.951733568s" totalImagesPullingTime="6.135647597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:56.950490628 +0000 UTC m=+55.940938668" watchObservedRunningTime="2026-07-08 02:58:56.951733568 +0000 UTC m=+55.942181605" Jul 08 02:58:56.970024 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:56.969976 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" podStartSLOduration=1.860423191 podStartE2EDuration="7.969956635s" totalImagesPullingTime="6.109533444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:56.969649601 +0000 UTC m=+55.960097640" watchObservedRunningTime="2026-07-08 02:58:56.969956635 +0000 UTC m=+55.960404674" Jul 08 02:58:57.035310 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.035268 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:57.093467 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.093435 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:57.165783 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.165735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:57.165941 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.165874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:58:57.165941 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.165924 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle podName:3a708c11-6608-4fd6-8f47-f2d9ded1569a nodeName:}" failed. No retries permitted until 2026-07-08 02:59:05.165900222 +0000 UTC m=+64.156348251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle") pod "router-default-5c46bc767-r4rqj" (UID: "3a708c11-6608-4fd6-8f47-f2d9ded1569a") : configmap references non-existent config key: service-ca.crt Jul 08 02:58:57.166076 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.166005 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 02:58:57.166124 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.166080 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs podName:3a708c11-6608-4fd6-8f47-f2d9ded1569a nodeName:}" failed. No retries permitted until 2026-07-08 02:59:05.166038586 +0000 UTC m=+64.156486618 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs") pod "router-default-5c46bc767-r4rqj" (UID: "3a708c11-6608-4fd6-8f47-f2d9ded1569a") : secret "router-metrics-certs-default" not found Jul 08 02:58:57.267157 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.267112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rrkzw\" (UID: \"5a53b345-6e70-44ee-b799-4f4c24c1d094\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" Jul 08 02:58:57.267335 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.267231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-4bs9z\" (UID: \"47662f67-2e4a-40ef-8cf9-a55730c0b6c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" Jul 08 02:58:57.267335 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.267309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:58:57.267443 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.267362 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 02:58:57.267502 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.267442 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls podName:47662f67-2e4a-40ef-8cf9-a55730c0b6c6 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:05.267419655 +0000 UTC m=+64.257867679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-4bs9z" (UID: "47662f67-2e4a-40ef-8cf9-a55730c0b6c6") : secret "samples-operator-tls" not found Jul 08 02:58:57.267502 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.267480 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 02:58:57.267612 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.267542 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls podName:cce8a6b5-0ecb-4ce6-bad9-e4387039a932 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:05.267528313 +0000 UTC m=+64.257976332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-pqdv5" (UID: "cce8a6b5-0ecb-4ce6-bad9-e4387039a932") : secret "cluster-monitoring-operator-tls" not found Jul 08 02:58:57.267783 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.267765 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 02:58:57.267915 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.267827 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert podName:5a53b345-6e70-44ee-b799-4f4c24c1d094 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:59.26779955 +0000 UTC m=+58.258247570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-rrkzw" (UID: "5a53b345-6e70-44ee-b799-4f4c24c1d094") : secret "networking-console-plugin-cert" not found Jul 08 02:58:57.298124 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.298066 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" podStartSLOduration=2.271160656 podStartE2EDuration="8.298025849s" totalImagesPullingTime="6.026865193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:56.991102624 +0000 UTC m=+55.981550661" watchObservedRunningTime="2026-07-08 02:58:57.298025849 +0000 UTC m=+56.288473887" Jul 08 02:58:57.298942 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.298904 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rz2g4"] Jul 08 02:58:57.301282 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.301262 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.305504 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.305385 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 02:58:57.305597 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.305580 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 02:58:57.305804 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.305787 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5d4df\"" Jul 08 02:58:57.316706 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.316686 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rz2g4"] Jul 08 02:58:57.341923 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.341895 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:57.368723 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.368687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-crio-socket\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.368861 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.368758 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rpdp\" (UniqueName: \"kubernetes.io/projected/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-api-access-7rpdp\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.368934 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.368861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.369066 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.369028 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.369151 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.369133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-data-volume\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.398972 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.398940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:57.470660 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.470566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-crio-socket\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.470660 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.470622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rpdp\" (UniqueName: \"kubernetes.io/projected/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-api-access-7rpdp\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.470883 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.470676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.470883 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.470704 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-crio-socket\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.470883 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.470782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.470883 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.470839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-data-volume\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.470883 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.470783 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 02:58:57.471162 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.470921 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm podName:5c7e8542-42da-4659-bccb-1b8bee7ec4d6 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:57.970901519 +0000 UTC m=+56.961349556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rz2g4" (UID: "5c7e8542-42da-4659-bccb-1b8bee7ec4d6") : configmap "kube-rbac-proxy" not found Jul 08 02:58:57.471162 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.470855 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 02:58:57.471162 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.471085 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls podName:5c7e8542-42da-4659-bccb-1b8bee7ec4d6 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:57.971070776 +0000 UTC m=+56.961518811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rz2g4" (UID: "5c7e8542-42da-4659-bccb-1b8bee7ec4d6") : secret "insights-runtime-extractor-tls" not found Jul 08 02:58:57.471322 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.471175 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-data-volume\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.484192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.484156 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rpdp\" (UniqueName: \"kubernetes.io/projected/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-api-access-7rpdp\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.902444 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.902362 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 02:58:57.902817 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.902760 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/0.log" Jul 08 02:58:57.902817 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.902793 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4220e41-4e35-4cc8-a67b-db04dd99e5cb" containerID="d65368b1dca97e9f164f1c7c209e79792bc5d80249cf6873ad4e35a881c13f8d" exitCode=255 Jul 08 02:58:57.902903 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.902877 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" event={"ID":"d4220e41-4e35-4cc8-a67b-db04dd99e5cb","Type":"ContainerDied","Data":"d65368b1dca97e9f164f1c7c209e79792bc5d80249cf6873ad4e35a881c13f8d"} Jul 08 02:58:57.902972 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.902961 2576 scope.go:117] "RemoveContainer" containerID="2b0b02ef5c98ee2417d05a0001b2558b04a1ff1e797ffb4b961786184e97632a" Jul 08 02:58:57.903273 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.903168 2576 scope.go:117] "RemoveContainer" containerID="d65368b1dca97e9f164f1c7c209e79792bc5d80249cf6873ad4e35a881c13f8d" Jul 08 02:58:57.903426 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.903394 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-lrzh5_openshift-console-operator(d4220e41-4e35-4cc8-a67b-db04dd99e5cb)\"" pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" podUID="d4220e41-4e35-4cc8-a67b-db04dd99e5cb" Jul 08 02:58:57.976407 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.976358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.976635 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.976612 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 02:58:57.976730 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.976695 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm podName:5c7e8542-42da-4659-bccb-1b8bee7ec4d6 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:58.976676628 +0000 UTC m=+57.967124661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rz2g4" (UID: "5c7e8542-42da-4659-bccb-1b8bee7ec4d6") : configmap "kube-rbac-proxy" not found Jul 08 02:58:57.976802 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:57.976731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:57.976844 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.976833 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 02:58:57.976906 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:57.976894 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls podName:5c7e8542-42da-4659-bccb-1b8bee7ec4d6 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:58.976878435 +0000 UTC m=+57.967326450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rz2g4" (UID: "5c7e8542-42da-4659-bccb-1b8bee7ec4d6") : secret "insights-runtime-extractor-tls" not found Jul 08 02:58:58.054785 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.054755 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:58.136281 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.136248 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-sfp2b"] Jul 08 02:58:58.138767 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.138743 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-sfp2b" Jul 08 02:58:58.141514 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.141493 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 08 02:58:58.141786 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.141772 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 08 02:58:58.141841 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.141814 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dqq5r\"" Jul 08 02:58:58.149426 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.149403 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-sfp2b"] Jul 08 02:58:58.157256 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.157195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:58.179349 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.179328 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nzbs\" (UniqueName: \"kubernetes.io/projected/7d9a4a38-9c77-49b5-b1e1-9f423d2e8bc8-kube-api-access-7nzbs\") pod \"migrator-d57d58cc9-sfp2b\" (UID: \"7d9a4a38-9c77-49b5-b1e1-9f423d2e8bc8\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-sfp2b" Jul 08 02:58:58.253424 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.253388 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:58.280133 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.280101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzbs\" (UniqueName: \"kubernetes.io/projected/7d9a4a38-9c77-49b5-b1e1-9f423d2e8bc8-kube-api-access-7nzbs\") pod \"migrator-d57d58cc9-sfp2b\" (UID: \"7d9a4a38-9c77-49b5-b1e1-9f423d2e8bc8\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-sfp2b" Jul 08 02:58:58.289183 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.289153 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nzbs\" (UniqueName: \"kubernetes.io/projected/7d9a4a38-9c77-49b5-b1e1-9f423d2e8bc8-kube-api-access-7nzbs\") pod \"migrator-d57d58cc9-sfp2b\" (UID: \"7d9a4a38-9c77-49b5-b1e1-9f423d2e8bc8\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-sfp2b" Jul 08 02:58:58.448087 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.448033 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-sfp2b" Jul 08 02:58:58.466788 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.466757 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:58.488189 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.488169 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h69hf_b53a5c28-a70a-4388-bb82-03cd3832d27a/dns-node-resolver/0.log" Jul 08 02:58:58.566413 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.566378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:58.575635 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.575615 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-sfp2b"] Jul 08 02:58:58.577919 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:58:58.577895 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9a4a38_9c77_49b5_b1e1_9f423d2e8bc8.slice/crio-751f8cebf4324459a7b3b1025f8cc1d85ebd2ad66e5fc22c4515d40712f659b6 WatchSource:0}: Error finding container 751f8cebf4324459a7b3b1025f8cc1d85ebd2ad66e5fc22c4515d40712f659b6: Status 404 returned error can't find the container with id 751f8cebf4324459a7b3b1025f8cc1d85ebd2ad66e5fc22c4515d40712f659b6 Jul 08 02:58:58.746469 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.746383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:58.845775 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.845747 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:58.907505 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.907477 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 02:58:58.907923 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.907821 2576 scope.go:117] "RemoveContainer" containerID="d65368b1dca97e9f164f1c7c209e79792bc5d80249cf6873ad4e35a881c13f8d" Jul 08 02:58:58.908024 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:58.908001 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-lrzh5_openshift-console-operator(d4220e41-4e35-4cc8-a67b-db04dd99e5cb)\"" pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" podUID="d4220e41-4e35-4cc8-a67b-db04dd99e5cb" Jul 08 02:58:58.908741 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.908720 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-sfp2b" event={"ID":"7d9a4a38-9c77-49b5-b1e1-9f423d2e8bc8","Type":"ContainerStarted","Data":"751f8cebf4324459a7b3b1025f8cc1d85ebd2ad66e5fc22c4515d40712f659b6"} Jul 08 02:58:58.985666 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.985629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:58.985849 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:58.985730 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 02:58:58.985849 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:58.985793 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm podName:5c7e8542-42da-4659-bccb-1b8bee7ec4d6 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:00.985774677 +0000 UTC m=+59.976222706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rz2g4" (UID: "5c7e8542-42da-4659-bccb-1b8bee7ec4d6") : configmap "kube-rbac-proxy" not found Jul 08 02:58:58.985849 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:58.985816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:58:58.986001 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:58.985963 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 02:58:58.986036 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:58.986025 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls podName:5c7e8542-42da-4659-bccb-1b8bee7ec4d6 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:00.986012763 +0000 UTC m=+59.976460783 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rz2g4" (UID: "5c7e8542-42da-4659-bccb-1b8bee7ec4d6") : secret "insights-runtime-extractor-tls" not found Jul 08 02:58:59.288152 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:59.288120 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rrkzw\" (UID: \"5a53b345-6e70-44ee-b799-4f4c24c1d094\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" Jul 08 02:58:59.288336 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:59.288323 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 02:58:59.288404 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:59.288392 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert podName:5a53b345-6e70-44ee-b799-4f4c24c1d094 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:03.288373882 +0000 UTC m=+62.278821909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-rrkzw" (UID: "5a53b345-6e70-44ee-b799-4f4c24c1d094") : secret "networking-console-plugin-cert" not found Jul 08 02:58:59.288504 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:59.288474 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-57xjh_97e0ac5b-7e47-49c6-a320-6f45f632c3d4/node-ca/0.log" Jul 08 02:58:59.471718 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:59.471681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:59.529239 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:59.529201 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:58:59.668935 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:59.668903 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:59.669126 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:59.668969 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:58:59.913559 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:59.913532 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-sfp2b" event={"ID":"7d9a4a38-9c77-49b5-b1e1-9f423d2e8bc8","Type":"ContainerStarted","Data":"a245e5213636ff5d41f2b8cec13c51530b3bea67abe332c70d5bcb090d58fc5d"} Jul 08 02:58:59.913941 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:58:59.913828 2576 scope.go:117] "RemoveContainer" containerID="d65368b1dca97e9f164f1c7c209e79792bc5d80249cf6873ad4e35a881c13f8d" Jul 08 02:58:59.914059 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:58:59.914019 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-lrzh5_openshift-console-operator(d4220e41-4e35-4cc8-a67b-db04dd99e5cb)\"" pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" podUID="d4220e41-4e35-4cc8-a67b-db04dd99e5cb" Jul 08 02:59:00.087863 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.087832 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lbwvc_cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03/serve-healthcheck-canary/0.log" Jul 08 02:59:00.380191 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.380112 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-lh7dc"] Jul 08 02:59:00.382069 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.382034 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-lh7dc" Jul 08 02:59:00.384889 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.384865 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 08 02:59:00.385521 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.385498 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 08 02:59:00.385639 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.385622 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vszld\"" Jul 08 02:59:00.386115 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.386095 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 08 02:59:00.386206 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.386120 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 08 02:59:00.395060 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.391444 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-lh7dc"] Jul 08 02:59:00.401959 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.401938 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:00.463420 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.463383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:00.499206 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.499176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/78598da1-8f01-447c-b0cc-54601fa29d47-signing-cabundle\") pod \"service-ca-758f599c5-lh7dc\" (UID: \"78598da1-8f01-447c-b0cc-54601fa29d47\") " pod="openshift-service-ca/service-ca-758f599c5-lh7dc" Jul 08 02:59:00.499365 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.499275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lssns\" (UniqueName: \"kubernetes.io/projected/78598da1-8f01-447c-b0cc-54601fa29d47-kube-api-access-lssns\") pod \"service-ca-758f599c5-lh7dc\" (UID: \"78598da1-8f01-447c-b0cc-54601fa29d47\") " pod="openshift-service-ca/service-ca-758f599c5-lh7dc" Jul 08 02:59:00.499414 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.499379 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/78598da1-8f01-447c-b0cc-54601fa29d47-signing-key\") pod \"service-ca-758f599c5-lh7dc\" (UID: \"78598da1-8f01-447c-b0cc-54601fa29d47\") " pod="openshift-service-ca/service-ca-758f599c5-lh7dc" Jul 08 02:59:00.564462 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.564424 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:00.599895 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.599870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/78598da1-8f01-447c-b0cc-54601fa29d47-signing-key\") pod \"service-ca-758f599c5-lh7dc\" (UID: \"78598da1-8f01-447c-b0cc-54601fa29d47\") " pod="openshift-service-ca/service-ca-758f599c5-lh7dc" Jul 08 02:59:00.600010 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.599921 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/78598da1-8f01-447c-b0cc-54601fa29d47-signing-cabundle\") pod \"service-ca-758f599c5-lh7dc\" (UID: \"78598da1-8f01-447c-b0cc-54601fa29d47\") " pod="openshift-service-ca/service-ca-758f599c5-lh7dc" Jul 08 02:59:00.600010 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.599983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lssns\" (UniqueName: \"kubernetes.io/projected/78598da1-8f01-447c-b0cc-54601fa29d47-kube-api-access-lssns\") pod \"service-ca-758f599c5-lh7dc\" (UID: \"78598da1-8f01-447c-b0cc-54601fa29d47\") " pod="openshift-service-ca/service-ca-758f599c5-lh7dc" Jul 08 02:59:00.600683 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.600666 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/78598da1-8f01-447c-b0cc-54601fa29d47-signing-cabundle\") pod \"service-ca-758f599c5-lh7dc\" (UID: \"78598da1-8f01-447c-b0cc-54601fa29d47\") " pod="openshift-service-ca/service-ca-758f599c5-lh7dc" Jul 08 02:59:00.602475 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.602457 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/78598da1-8f01-447c-b0cc-54601fa29d47-signing-key\") pod \"service-ca-758f599c5-lh7dc\" (UID: \"78598da1-8f01-447c-b0cc-54601fa29d47\") " pod="openshift-service-ca/service-ca-758f599c5-lh7dc" Jul 08 02:59:00.609521 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.609495 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lssns\" (UniqueName: \"kubernetes.io/projected/78598da1-8f01-447c-b0cc-54601fa29d47-kube-api-access-lssns\") pod \"service-ca-758f599c5-lh7dc\" (UID: \"78598da1-8f01-447c-b0cc-54601fa29d47\") " pod="openshift-service-ca/service-ca-758f599c5-lh7dc" Jul 08 02:59:00.690503 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.690483 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-lh7dc" Jul 08 02:59:00.813120 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.813091 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-lh7dc"] Jul 08 02:59:00.816159 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:00.816132 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78598da1_8f01_447c_b0cc_54601fa29d47.slice/crio-53f05992f46115d4fa48cbe79f7a104eecb526a6634a834e4a89acd1d36dab29 WatchSource:0}: Error finding container 53f05992f46115d4fa48cbe79f7a104eecb526a6634a834e4a89acd1d36dab29: Status 404 returned error can't find the container with id 53f05992f46115d4fa48cbe79f7a104eecb526a6634a834e4a89acd1d36dab29 Jul 08 02:59:00.920159 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.920109 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-sfp2b" event={"ID":"7d9a4a38-9c77-49b5-b1e1-9f423d2e8bc8","Type":"ContainerStarted","Data":"bd6278b37d438cb763192fd10c4d34298928d0d450795e504fc0e80729b5868e"} Jul 08 02:59:00.921497 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.921472 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-lh7dc" event={"ID":"78598da1-8f01-447c-b0cc-54601fa29d47","Type":"ContainerStarted","Data":"6cbcbaeaaa0e3c3bc3ca745629dc0575fd9efabc188d96e6ca829b7559a99025"} Jul 08 02:59:00.921614 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.921516 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-lh7dc" event={"ID":"78598da1-8f01-447c-b0cc-54601fa29d47","Type":"ContainerStarted","Data":"53f05992f46115d4fa48cbe79f7a104eecb526a6634a834e4a89acd1d36dab29"} Jul 08 02:59:00.941575 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.941485 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-sfp2b" podStartSLOduration=1.694797299 podStartE2EDuration="2.941468794s" totalImagesPullingTime="1.246671495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:00.94091195 +0000 UTC m=+59.931359987" watchObservedRunningTime="2026-07-08 02:59:00.941468794 +0000 UTC m=+59.931916830" Jul 08 02:59:00.961087 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:00.961029 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-lh7dc" podStartSLOduration=0.961015146 podStartE2EDuration="961.015146ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:00.959209207 +0000 UTC m=+59.949657244" watchObservedRunningTime="2026-07-08 02:59:00.961015146 +0000 UTC m=+59.951463185" Jul 08 02:59:01.004072 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:01.004022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:59:01.004221 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:01.004132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:59:01.004221 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:01.004141 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 02:59:01.004221 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:01.004214 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm podName:5c7e8542-42da-4659-bccb-1b8bee7ec4d6 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:05.004194142 +0000 UTC m=+63.994642173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rz2g4" (UID: "5c7e8542-42da-4659-bccb-1b8bee7ec4d6") : configmap "kube-rbac-proxy" not found Jul 08 02:59:01.004368 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:01.004256 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 02:59:01.004368 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:01.004323 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls podName:5c7e8542-42da-4659-bccb-1b8bee7ec4d6 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:05.00430913 +0000 UTC m=+63.994757144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rz2g4" (UID: "5c7e8542-42da-4659-bccb-1b8bee7ec4d6") : secret "insights-runtime-extractor-tls" not found Jul 08 02:59:01.221013 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:01.220923 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:01.657911 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:01.657814 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:01.786315 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:01.786273 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qb87" Jul 08 02:59:01.847764 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:01.847734 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:01.931411 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:01.931318 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:02.030987 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:02.030952 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:02.134455 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:02.134402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:02.270125 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:02.270023 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:02.359183 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:02.359148 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:03.148624 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:03.148591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:03.327022 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:03.326992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rrkzw\" (UID: \"5a53b345-6e70-44ee-b799-4f4c24c1d094\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" Jul 08 02:59:03.327181 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:03.327160 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 02:59:03.327243 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:03.327232 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert podName:5a53b345-6e70-44ee-b799-4f4c24c1d094 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:11.327211697 +0000 UTC m=+70.317659717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-rrkzw" (UID: "5a53b345-6e70-44ee-b799-4f4c24c1d094") : secret "networking-console-plugin-cert" not found Jul 08 02:59:03.743909 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:03.743864 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:03.949883 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:03.949849 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:04.162482 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:04.162451 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:04.351273 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:04.351227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:04.422878 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:04.422784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:04.479172 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:04.479135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:05.019819 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.019781 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:05.044871 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.044847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:59:05.045005 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.044987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:59:05.045070 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:05.044994 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 02:59:05.045145 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:05.045111 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls podName:5c7e8542-42da-4659-bccb-1b8bee7ec4d6 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:13.04509784 +0000 UTC m=+72.035545855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rz2g4" (UID: "5c7e8542-42da-4659-bccb-1b8bee7ec4d6") : secret "insights-runtime-extractor-tls" not found Jul 08 02:59:05.045145 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:05.045035 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 02:59:05.045251 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:05.045207 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm podName:5c7e8542-42da-4659-bccb-1b8bee7ec4d6 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:13.045186134 +0000 UTC m=+72.035634152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rz2g4" (UID: "5c7e8542-42da-4659-bccb-1b8bee7ec4d6") : configmap "kube-rbac-proxy" not found Jul 08 02:59:05.161145 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.161104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:05.246834 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.246799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:59:05.247217 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.246913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:59:05.247217 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:05.246983 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle podName:3a708c11-6608-4fd6-8f47-f2d9ded1569a nodeName:}" failed. No retries permitted until 2026-07-08 02:59:21.246965073 +0000 UTC m=+80.237413108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle") pod "router-default-5c46bc767-r4rqj" (UID: "3a708c11-6608-4fd6-8f47-f2d9ded1569a") : configmap references non-existent config key: service-ca.crt Jul 08 02:59:05.247217 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:05.247035 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 02:59:05.247217 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:05.247101 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs podName:3a708c11-6608-4fd6-8f47-f2d9ded1569a nodeName:}" failed. No retries permitted until 2026-07-08 02:59:21.247088624 +0000 UTC m=+80.237536654 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs") pod "router-default-5c46bc767-r4rqj" (UID: "3a708c11-6608-4fd6-8f47-f2d9ded1569a") : secret "router-metrics-certs-default" not found Jul 08 02:59:05.347672 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.347593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:59:05.347818 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.347719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-4bs9z\" (UID: \"47662f67-2e4a-40ef-8cf9-a55730c0b6c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" Jul 08 02:59:05.347818 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:05.347746 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 02:59:05.347818 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:05.347806 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls podName:cce8a6b5-0ecb-4ce6-bad9-e4387039a932 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:21.347789709 +0000 UTC m=+80.338237729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-pqdv5" (UID: "cce8a6b5-0ecb-4ce6-bad9-e4387039a932") : secret "cluster-monitoring-operator-tls" not found Jul 08 02:59:05.349559 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.349526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:05.350466 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.350447 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47662f67-2e4a-40ef-8cf9-a55730c0b6c6-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-4bs9z\" (UID: \"47662f67-2e4a-40ef-8cf9-a55730c0b6c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" Jul 08 02:59:05.379378 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.379357 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tvjqr\"" Jul 08 02:59:05.387720 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.387701 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" Jul 08 02:59:05.424104 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.424072 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:05.516889 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.516848 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z"] Jul 08 02:59:05.746591 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.746556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:05.939850 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.939810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" event={"ID":"47662f67-2e4a-40ef-8cf9-a55730c0b6c6","Type":"ContainerStarted","Data":"b73be2bcdbb14be30a0e11a8003b2dd9a1178eff006be7e30fb73a11ae9157ab"} Jul 08 02:59:05.951076 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:05.951027 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:06.010493 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.010401 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:06.155575 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.155536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:06.258826 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.258787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs\") pod \"network-metrics-daemon-wdvzz\" (UID: \"c392372d-4d45-4636-8058-57fd87930e8e\") " pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 02:59:06.264092 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.264018 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 02:59:06.269123 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:06.269106 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 02:59:06.269187 ip-10-0-142-99 kubenswrapper[2576]: E0708 02:59:06.269176 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs podName:c392372d-4d45-4636-8058-57fd87930e8e nodeName:}" failed. No retries permitted until 2026-07-08 03:00:10.269146623 +0000 UTC m=+129.259594637 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs") pod "network-metrics-daemon-wdvzz" (UID: "c392372d-4d45-4636-8058-57fd87930e8e") : secret "metrics-daemon-secret" not found Jul 08 02:59:06.360823 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.360779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:59:06.360990 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.360940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tj2\" (UniqueName: \"kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2\") pod \"network-check-target-bwz4j\" (UID: \"29273a5e-bb56-4d02-9905-33accc2373ef\") " pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:59:06.367035 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.367006 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4tj2\" (UniqueName: \"kubernetes.io/projected/29273a5e-bb56-4d02-9905-33accc2373ef-kube-api-access-q4tj2\") pod \"network-check-target-bwz4j\" (UID: \"29273a5e-bb56-4d02-9905-33accc2373ef\") " pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:59:06.369164 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.369142 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 02:59:06.378846 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.378819 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0343e166-1101-45b0-a57e-cc26b4858985-original-pull-secret\") pod \"global-pull-secret-syncer-78f5d\" (UID: \"0343e166-1101-45b0-a57e-cc26b4858985\") " pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:59:06.549648 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.549551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:06.638402 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.638374 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v7mdx\"" Jul 08 02:59:06.645592 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.645560 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:59:06.651346 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.651326 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78f5d" Jul 08 02:59:06.795912 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.795888 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bwz4j"] Jul 08 02:59:06.799006 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:06.798977 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29273a5e_bb56_4d02_9905_33accc2373ef.slice/crio-c0d30cee7176427c8f66afd3d56d15360cea37bfacbef19ff6b7bd6abe4b2dbc WatchSource:0}: Error finding container c0d30cee7176427c8f66afd3d56d15360cea37bfacbef19ff6b7bd6abe4b2dbc: Status 404 returned error can't find the container with id c0d30cee7176427c8f66afd3d56d15360cea37bfacbef19ff6b7bd6abe4b2dbc Jul 08 02:59:06.816343 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.816319 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-78f5d"] Jul 08 02:59:06.820575 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:06.820547 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0343e166_1101_45b0_a57e_cc26b4858985.slice/crio-b25769e57ae21311b3ba7f939367d98a829c12e0f5ceea63bd190df181887b2f WatchSource:0}: Error finding container b25769e57ae21311b3ba7f939367d98a829c12e0f5ceea63bd190df181887b2f: Status 404 returned error can't find the container with id b25769e57ae21311b3ba7f939367d98a829c12e0f5ceea63bd190df181887b2f Jul 08 02:59:06.944840 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.944804 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-78f5d" event={"ID":"0343e166-1101-45b0-a57e-cc26b4858985","Type":"ContainerStarted","Data":"b25769e57ae21311b3ba7f939367d98a829c12e0f5ceea63bd190df181887b2f"} Jul 08 02:59:06.946882 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.946850 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bwz4j" event={"ID":"29273a5e-bb56-4d02-9905-33accc2373ef","Type":"ContainerStarted","Data":"33fcc520827d8fe585153cda568840b8852b1cd8d1c814c7c0a5de1d1bd986d3"} Jul 08 02:59:06.947021 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.946896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bwz4j" event={"ID":"29273a5e-bb56-4d02-9905-33accc2373ef","Type":"ContainerStarted","Data":"c0d30cee7176427c8f66afd3d56d15360cea37bfacbef19ff6b7bd6abe4b2dbc"} Jul 08 02:59:06.947021 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.946973 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:59:06.969544 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:06.969513 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:07.556835 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:07.556802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:07.778623 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:07.778575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:59:07.781709 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:07.781680 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls\") pod \"image-registry-6c4cc5dff5-lx8g9\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:59:07.854286 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:07.854213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:07.879711 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:07.879676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:59:07.882463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:07.882425 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94c47594-a484-4654-8cae-7325c0d9d581-metrics-tls\") pod \"dns-default-6j82n\" (UID: \"94c47594-a484-4654-8cae-7325c0d9d581\") " pod="openshift-dns/dns-default-6j82n" Jul 08 02:59:07.953303 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:07.953274 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" event={"ID":"47662f67-2e4a-40ef-8cf9-a55730c0b6c6","Type":"ContainerStarted","Data":"5b6cfa05d6df90555f0972747e70c82bae24935dec13a7d1e61131dfa8eb1440"} Jul 08 02:59:07.953451 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:07.953314 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" event={"ID":"47662f67-2e4a-40ef-8cf9-a55730c0b6c6","Type":"ContainerStarted","Data":"c9c2493aacd9d4e6400c173cd9e5aab78333f5e1530f40048ca28322d48e5b25"} Jul 08 02:59:07.975556 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:07.975497 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bwz4j" podStartSLOduration=66.975477858 podStartE2EDuration="1m6.975477858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:06.964140564 +0000 UTC m=+65.954588617" watchObservedRunningTime="2026-07-08 02:59:07.975477858 +0000 UTC m=+66.965925899" Jul 08 02:59:07.975918 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:07.975886 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-4bs9z" podStartSLOduration=16.973350768 podStartE2EDuration="18.975878667s" totalImagesPullingTime="2.002527899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:07.974199214 +0000 UTC m=+66.964647253" watchObservedRunningTime="2026-07-08 02:59:07.975878667 +0000 UTC m=+66.966326705" Jul 08 02:59:08.008089 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:08.008065 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xcxgk\"" Jul 08 02:59:08.015655 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:08.015620 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:59:08.047397 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:08.047375 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6djh2\"" Jul 08 02:59:08.054712 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:08.054684 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6j82n" Jul 08 02:59:08.184604 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:08.184572 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c4cc5dff5-lx8g9"] Jul 08 02:59:08.189176 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:08.189148 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f7b270_4a1d_4fbb_9913_56cc0e4fd1b5.slice/crio-21507af95ea6410ec283a10dd796d90c7d2c882cefdba6afccecbed9987dc129 WatchSource:0}: Error finding container 21507af95ea6410ec283a10dd796d90c7d2c882cefdba6afccecbed9987dc129: Status 404 returned error can't find the container with id 21507af95ea6410ec283a10dd796d90c7d2c882cefdba6afccecbed9987dc129 Jul 08 02:59:08.216686 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:08.216661 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6j82n"] Jul 08 02:59:08.223195 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:08.223169 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c47594_a484_4654_8cae_7325c0d9d581.slice/crio-86db4a8849fc7f4239599c7abe662bba954396de755d7628ad4049369fa4b67a WatchSource:0}: Error finding container 86db4a8849fc7f4239599c7abe662bba954396de755d7628ad4049369fa4b67a: Status 404 returned error can't find the container with id 86db4a8849fc7f4239599c7abe662bba954396de755d7628ad4049369fa4b67a Jul 08 02:59:08.958434 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:08.958397 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6j82n" event={"ID":"94c47594-a484-4654-8cae-7325c0d9d581","Type":"ContainerStarted","Data":"86db4a8849fc7f4239599c7abe662bba954396de755d7628ad4049369fa4b67a"} Jul 08 02:59:08.960999 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:08.960250 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" event={"ID":"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5","Type":"ContainerStarted","Data":"e068cb661c14c5bfcb9da2bc2e364e04e7ebd217a1b5f5112c8b0d7508a4ea0c"} Jul 08 02:59:08.960999 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:08.960305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" event={"ID":"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5","Type":"ContainerStarted","Data":"21507af95ea6410ec283a10dd796d90c7d2c882cefdba6afccecbed9987dc129"} Jul 08 02:59:08.960999 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:08.960749 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:59:08.981331 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:08.981283 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" podStartSLOduration=66.98126458 podStartE2EDuration="1m6.98126458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:08.980445014 +0000 UTC m=+67.970893051" watchObservedRunningTime="2026-07-08 02:59:08.98126458 +0000 UTC m=+67.971712618" Jul 08 02:59:11.414399 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:11.414361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rrkzw\" (UID: \"5a53b345-6e70-44ee-b799-4f4c24c1d094\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" Jul 08 02:59:11.417299 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:11.417271 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a53b345-6e70-44ee-b799-4f4c24c1d094-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rrkzw\" (UID: \"5a53b345-6e70-44ee-b799-4f4c24c1d094\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" Jul 08 02:59:11.521172 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:11.521138 2576 scope.go:117] "RemoveContainer" containerID="d65368b1dca97e9f164f1c7c209e79792bc5d80249cf6873ad4e35a881c13f8d" Jul 08 02:59:11.554415 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:11.554375 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ck74d\"" Jul 08 02:59:11.561842 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:11.561818 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" Jul 08 02:59:12.223929 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:12.223864 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw"] Jul 08 02:59:12.229936 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:12.229908 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a53b345_6e70_44ee_b799_4f4c24c1d094.slice/crio-7b19f313a1f645e70f48c46179d1444038c0f2f4daed3fa51252c218559fd57e WatchSource:0}: Error finding container 7b19f313a1f645e70f48c46179d1444038c0f2f4daed3fa51252c218559fd57e: Status 404 returned error can't find the container with id 7b19f313a1f645e70f48c46179d1444038c0f2f4daed3fa51252c218559fd57e Jul 08 02:59:12.983901 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:12.983861 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-78f5d" event={"ID":"0343e166-1101-45b0-a57e-cc26b4858985","Type":"ContainerStarted","Data":"32a02b293c76ec94667b886ac2fd74fb6fb5c9a1bb2365be448cd85e1873a53b"} Jul 08 02:59:12.986371 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:12.986341 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6j82n" event={"ID":"94c47594-a484-4654-8cae-7325c0d9d581","Type":"ContainerStarted","Data":"2b766b192d376b9fecea5b301bd961453265d08ad43fc79b2a1dd8386c92abc4"} Jul 08 02:59:12.986525 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:12.986413 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6j82n" event={"ID":"94c47594-a484-4654-8cae-7325c0d9d581","Type":"ContainerStarted","Data":"7e720961770cbeed189457391b2cdcb91b959a9a6884563419d404fa0332e5d8"} Jul 08 02:59:12.986525 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:12.986446 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6j82n" Jul 08 02:59:12.988636 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:12.988619 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 02:59:12.988723 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:12.988695 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" event={"ID":"d4220e41-4e35-4cc8-a67b-db04dd99e5cb","Type":"ContainerStarted","Data":"36a89eb2488ac862fc3e0dcb9543e459eb3c0d2c108b1b93d669a6936260bc7f"} Jul 08 02:59:12.989100 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:12.989080 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:59:12.990887 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:12.990865 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" event={"ID":"5a53b345-6e70-44ee-b799-4f4c24c1d094","Type":"ContainerStarted","Data":"7b19f313a1f645e70f48c46179d1444038c0f2f4daed3fa51252c218559fd57e"} Jul 08 02:59:12.995308 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:12.995290 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" Jul 08 02:59:13.007463 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:13.007417 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-78f5d" podStartSLOduration=65.743947727 podStartE2EDuration="1m11.007405656s" totalImagesPullingTime="5.263457929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:13.005715524 +0000 UTC m=+71.996163562" watchObservedRunningTime="2026-07-08 02:59:13.007405656 +0000 UTC m=+71.997853694" Jul 08 02:59:13.030158 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:13.030110 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6j82n" podStartSLOduration=34.18147768 podStartE2EDuration="38.030095706s" totalImagesPullingTime="3.848618026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:13.029739943 +0000 UTC m=+72.020187981" watchObservedRunningTime="2026-07-08 02:59:13.030095706 +0000 UTC m=+72.020543744" Jul 08 02:59:13.074596 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:13.074540 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-lrzh5" podStartSLOduration=17.971395758 podStartE2EDuration="24.074521145s" totalImagesPullingTime="6.103125387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:13.073875372 +0000 UTC m=+72.064323410" watchObservedRunningTime="2026-07-08 02:59:13.074521145 +0000 UTC m=+72.064969182" Jul 08 02:59:13.131296 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:13.131256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:59:13.131533 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:13.131505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:59:13.132142 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:13.132094 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:59:13.134172 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:13.134149 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c7e8542-42da-4659-bccb-1b8bee7ec4d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rz2g4\" (UID: \"5c7e8542-42da-4659-bccb-1b8bee7ec4d6\") " pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:59:13.215361 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:13.215328 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5d4df\"" Jul 08 02:59:13.222988 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:13.222950 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rz2g4" Jul 08 02:59:13.566394 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:13.566365 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rz2g4"] Jul 08 02:59:13.569318 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:13.569293 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7e8542_42da_4659_bccb_1b8bee7ec4d6.slice/crio-10ff0cab2300ec90629d61c7086304a9e455d938e8c3fd56aa3afe41a538e0da WatchSource:0}: Error finding container 10ff0cab2300ec90629d61c7086304a9e455d938e8c3fd56aa3afe41a538e0da: Status 404 returned error can't find the container with id 10ff0cab2300ec90629d61c7086304a9e455d938e8c3fd56aa3afe41a538e0da Jul 08 02:59:14.000002 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:13.999740 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" event={"ID":"5a53b345-6e70-44ee-b799-4f4c24c1d094","Type":"ContainerStarted","Data":"97df7dff326719c3f1b21ae8f8bd2e5810dcc6f4d014fa279866f81afbee490a"} Jul 08 02:59:14.002671 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:14.002646 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rz2g4" event={"ID":"5c7e8542-42da-4659-bccb-1b8bee7ec4d6","Type":"ContainerStarted","Data":"10ff0cab2300ec90629d61c7086304a9e455d938e8c3fd56aa3afe41a538e0da"} Jul 08 02:59:14.019224 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:14.018266 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rrkzw" podStartSLOduration=17.792912213 podStartE2EDuration="19.018246018s" totalImagesPullingTime="1.225333805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:14.017340247 +0000 UTC m=+73.007788285" watchObservedRunningTime="2026-07-08 02:59:14.018246018 +0000 UTC m=+73.008694074" Jul 08 02:59:18.019709 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:18.019675 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rz2g4" event={"ID":"5c7e8542-42da-4659-bccb-1b8bee7ec4d6","Type":"ContainerStarted","Data":"5da8679bb4576f1724c01ca221f2cf86d73b85a209e83caec0dd993b6aeae59a"} Jul 08 02:59:18.020104 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:18.019737 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rz2g4" event={"ID":"5c7e8542-42da-4659-bccb-1b8bee7ec4d6","Type":"ContainerStarted","Data":"263bc1a58698e20a6679354756fe38078ce291aa4b479ee05b9a75ee142db7bf"} Jul 08 02:59:21.033199 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:21.033163 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rz2g4" event={"ID":"5c7e8542-42da-4659-bccb-1b8bee7ec4d6","Type":"ContainerStarted","Data":"ed5e37aaa00ab082ba005ae273d75589d24532d6d327371d2c12afb2c2251467"} Jul 08 02:59:21.053810 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:21.053759 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rz2g4" podStartSLOduration=17.340692401 podStartE2EDuration="24.05374389s" totalImagesPullingTime="6.713051489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:21.052405334 +0000 UTC m=+80.042853371" watchObservedRunningTime="2026-07-08 02:59:21.05374389 +0000 UTC m=+80.044191927" Jul 08 02:59:21.310873 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:21.310765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:59:21.310873 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:21.310836 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:59:21.311449 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:21.311423 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a708c11-6608-4fd6-8f47-f2d9ded1569a-service-ca-bundle\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:59:21.313227 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:21.313202 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a708c11-6608-4fd6-8f47-f2d9ded1569a-metrics-certs\") pod \"router-default-5c46bc767-r4rqj\" (UID: \"3a708c11-6608-4fd6-8f47-f2d9ded1569a\") " pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:59:21.412104 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:21.412032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:59:21.414527 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:21.414497 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cce8a6b5-0ecb-4ce6-bad9-e4387039a932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pqdv5\" (UID: \"cce8a6b5-0ecb-4ce6-bad9-e4387039a932\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:59:21.489293 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:21.489264 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mtkbk\"" Jul 08 02:59:21.496629 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:21.496599 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:59:21.615155 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:21.615127 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-h2vn8\"" Jul 08 02:59:21.623537 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:21.623506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" Jul 08 02:59:21.632594 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:21.632563 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c46bc767-r4rqj"] Jul 08 02:59:21.636506 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:21.636475 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a708c11_6608_4fd6_8f47_f2d9ded1569a.slice/crio-70dfd22ededdf827fa6f1be02089fdabc968a35fe81f67b09ba0cb1b0109a465 WatchSource:0}: Error finding container 70dfd22ededdf827fa6f1be02089fdabc968a35fe81f67b09ba0cb1b0109a465: Status 404 returned error can't find the container with id 70dfd22ededdf827fa6f1be02089fdabc968a35fe81f67b09ba0cb1b0109a465 Jul 08 02:59:21.770161 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:21.770131 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5"] Jul 08 02:59:21.773381 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:21.773356 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce8a6b5_0ecb_4ce6_bad9_e4387039a932.slice/crio-91db3dd57232ad508a5dba510d8e2b46f469bf64b98cadacfa15f3aaad534315 WatchSource:0}: Error finding container 91db3dd57232ad508a5dba510d8e2b46f469bf64b98cadacfa15f3aaad534315: Status 404 returned error can't find the container with id 91db3dd57232ad508a5dba510d8e2b46f469bf64b98cadacfa15f3aaad534315 Jul 08 02:59:22.038111 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:22.038071 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c46bc767-r4rqj" event={"ID":"3a708c11-6608-4fd6-8f47-f2d9ded1569a","Type":"ContainerStarted","Data":"0de0c0b0fb75c81a40efcb62b61a90720af1507f3f003b3857d906aae992b124"} Jul 08 02:59:22.038596 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:22.038131 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c46bc767-r4rqj" event={"ID":"3a708c11-6608-4fd6-8f47-f2d9ded1569a","Type":"ContainerStarted","Data":"70dfd22ededdf827fa6f1be02089fdabc968a35fe81f67b09ba0cb1b0109a465"} Jul 08 02:59:22.039324 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:22.039287 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" event={"ID":"cce8a6b5-0ecb-4ce6-bad9-e4387039a932","Type":"ContainerStarted","Data":"91db3dd57232ad508a5dba510d8e2b46f469bf64b98cadacfa15f3aaad534315"} Jul 08 02:59:22.057336 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:22.057289 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5c46bc767-r4rqj" podStartSLOduration=33.057273749 podStartE2EDuration="33.057273749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:22.056133132 +0000 UTC m=+81.046581171" watchObservedRunningTime="2026-07-08 02:59:22.057273749 +0000 UTC m=+81.047721783" Jul 08 02:59:22.497787 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:22.497742 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:59:22.500390 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:22.500364 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:59:23.006338 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:23.006307 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6j82n" Jul 08 02:59:23.043525 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:23.043492 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:59:23.044836 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:23.044815 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5c46bc767-r4rqj" Jul 08 02:59:24.049727 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:24.049631 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" event={"ID":"cce8a6b5-0ecb-4ce6-bad9-e4387039a932","Type":"ContainerStarted","Data":"a60d17099fc2b7ff1185a8e14b485d2160fd10c3f4aca9151b518f2cc76d86f3"} Jul 08 02:59:24.074133 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:24.074086 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pqdv5" podStartSLOduration=33.083235524 podStartE2EDuration="35.07406879s" totalImagesPullingTime="1.990833266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:24.072817789 +0000 UTC m=+83.063265826" watchObservedRunningTime="2026-07-08 02:59:24.07406879 +0000 UTC m=+83.064516852" Jul 08 02:59:25.118111 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.118036 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb"] Jul 08 02:59:25.121633 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.121612 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p"] Jul 08 02:59:25.121758 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.121743 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb" Jul 08 02:59:25.124971 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.124723 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 08 02:59:25.124971 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.124757 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" Jul 08 02:59:25.124971 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.124739 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 08 02:59:25.125856 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.125832 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-h679l\"" Jul 08 02:59:25.126413 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.125993 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 08 02:59:25.126413 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.125862 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 08 02:59:25.129839 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.129817 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 08 02:59:25.132276 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.132254 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk"] Jul 08 02:59:25.135428 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.135411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.138524 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.138504 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 08 02:59:25.138791 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.138765 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 08 02:59:25.138955 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.138895 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 08 02:59:25.139216 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.139201 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 08 02:59:25.140382 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.140363 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb"] Jul 08 02:59:25.141459 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.141441 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p"] Jul 08 02:59:25.152818 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.152793 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk"] Jul 08 02:59:25.195893 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.195866 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-cgq8z"] Jul 08 02:59:25.199104 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.199087 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-cgq8z" Jul 08 02:59:25.201873 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.201855 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5tx9k\"" Jul 08 02:59:25.202324 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.202303 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 02:59:25.202492 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.202473 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-64bj8"] Jul 08 02:59:25.202659 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.202642 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 02:59:25.205406 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.205392 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-64bj8" Jul 08 02:59:25.207756 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.207737 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-k4vk7\"" Jul 08 02:59:25.208014 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.208000 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 08 02:59:25.217730 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.217681 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-cgq8z"] Jul 08 02:59:25.233619 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.233592 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-64bj8"] Jul 08 02:59:25.246612 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.246587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7a6d5cc2-87d6-4cc7-9711-349a152ff665-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.246734 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.246622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24nh9\" (UniqueName: \"kubernetes.io/projected/f4413f61-b3e4-4caa-84ab-0c0644741cf7-kube-api-access-24nh9\") pod \"klusterlet-addon-workmgr-84f4b8c98f-vwg4p\" (UID: \"f4413f61-b3e4-4caa-84ab-0c0644741cf7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" Jul 08 02:59:25.246734 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.246640 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7a6d5cc2-87d6-4cc7-9711-349a152ff665-ca\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.246734 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.246680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a6d5cc2-87d6-4cc7-9711-349a152ff665-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.246734 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.246726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4413f61-b3e4-4caa-84ab-0c0644741cf7-tmp\") pod \"klusterlet-addon-workmgr-84f4b8c98f-vwg4p\" (UID: \"f4413f61-b3e4-4caa-84ab-0c0644741cf7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" Jul 08 02:59:25.246920 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.246849 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj68v\" (UniqueName: \"kubernetes.io/projected/6aff4eef-9c66-4045-855c-08b914b03135-kube-api-access-vj68v\") pod \"managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb\" (UID: \"6aff4eef-9c66-4045-855c-08b914b03135\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb" Jul 08 02:59:25.247010 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.246990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7a6d5cc2-87d6-4cc7-9711-349a152ff665-hub\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.247092 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.247070 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6aff4eef-9c66-4045-855c-08b914b03135-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb\" (UID: \"6aff4eef-9c66-4045-855c-08b914b03135\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb" Jul 08 02:59:25.247154 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.247105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6d5cc2-87d6-4cc7-9711-349a152ff665-additional-service-ca\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.247154 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.247127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f4413f61-b3e4-4caa-84ab-0c0644741cf7-klusterlet-config\") pod \"klusterlet-addon-workmgr-84f4b8c98f-vwg4p\" (UID: \"f4413f61-b3e4-4caa-84ab-0c0644741cf7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" Jul 08 02:59:25.247154 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.247144 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-778j8\" (UniqueName: \"kubernetes.io/projected/7a6d5cc2-87d6-4cc7-9711-349a152ff665-kube-api-access-778j8\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.345689 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.345615 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8f8c64965-gjw9f"] Jul 08 02:59:25.348444 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.348417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7a6d5cc2-87d6-4cc7-9711-349a152ff665-hub\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.348571 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.348457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6aff4eef-9c66-4045-855c-08b914b03135-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb\" (UID: \"6aff4eef-9c66-4045-855c-08b914b03135\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb" Jul 08 02:59:25.348571 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.348482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6d5cc2-87d6-4cc7-9711-349a152ff665-additional-service-ca\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.348571 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.348517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f4413f61-b3e4-4caa-84ab-0c0644741cf7-klusterlet-config\") pod \"klusterlet-addon-workmgr-84f4b8c98f-vwg4p\" (UID: \"f4413f61-b3e4-4caa-84ab-0c0644741cf7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" Jul 08 02:59:25.348571 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.348539 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-778j8\" (UniqueName: \"kubernetes.io/projected/7a6d5cc2-87d6-4cc7-9711-349a152ff665-kube-api-access-778j8\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.348786 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.348571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7a6d5cc2-87d6-4cc7-9711-349a152ff665-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.348786 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.348603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24nh9\" (UniqueName: \"kubernetes.io/projected/f4413f61-b3e4-4caa-84ab-0c0644741cf7-kube-api-access-24nh9\") pod \"klusterlet-addon-workmgr-84f4b8c98f-vwg4p\" (UID: \"f4413f61-b3e4-4caa-84ab-0c0644741cf7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" Jul 08 02:59:25.348786 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.348628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7a6d5cc2-87d6-4cc7-9711-349a152ff665-ca\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.348786 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.348675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a6d5cc2-87d6-4cc7-9711-349a152ff665-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.348786 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.348706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e78201e6-cbb1-4203-a1f4-103c40ceb783-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-64bj8\" (UID: \"e78201e6-cbb1-4203-a1f4-103c40ceb783\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-64bj8" Jul 08 02:59:25.348786 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.348746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4413f61-b3e4-4caa-84ab-0c0644741cf7-tmp\") pod \"klusterlet-addon-workmgr-84f4b8c98f-vwg4p\" (UID: \"f4413f61-b3e4-4caa-84ab-0c0644741cf7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" Jul 08 02:59:25.349243 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.348802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj68v\" (UniqueName: \"kubernetes.io/projected/6aff4eef-9c66-4045-855c-08b914b03135-kube-api-access-vj68v\") pod \"managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb\" (UID: \"6aff4eef-9c66-4045-855c-08b914b03135\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb" Jul 08 02:59:25.349243 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.348843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k622k\" (UniqueName: \"kubernetes.io/projected/819aac06-7339-40a7-b54d-cffb09e62092-kube-api-access-k622k\") pod \"downloads-685bfb5f64-cgq8z\" (UID: \"819aac06-7339-40a7-b54d-cffb09e62092\") " pod="openshift-console/downloads-685bfb5f64-cgq8z" Jul 08 02:59:25.349243 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.349079 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.349466 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.349409 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4413f61-b3e4-4caa-84ab-0c0644741cf7-tmp\") pod \"klusterlet-addon-workmgr-84f4b8c98f-vwg4p\" (UID: \"f4413f61-b3e4-4caa-84ab-0c0644741cf7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" Jul 08 02:59:25.350948 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.350921 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6d5cc2-87d6-4cc7-9711-349a152ff665-additional-service-ca\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.351895 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.351869 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7a6d5cc2-87d6-4cc7-9711-349a152ff665-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.351995 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.351973 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7a6d5cc2-87d6-4cc7-9711-349a152ff665-ca\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.352094 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.351998 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6aff4eef-9c66-4045-855c-08b914b03135-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb\" (UID: \"6aff4eef-9c66-4045-855c-08b914b03135\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb" Jul 08 02:59:25.352234 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.352211 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7a6d5cc2-87d6-4cc7-9711-349a152ff665-hub\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.352472 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.352452 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f4413f61-b3e4-4caa-84ab-0c0644741cf7-klusterlet-config\") pod \"klusterlet-addon-workmgr-84f4b8c98f-vwg4p\" (UID: \"f4413f61-b3e4-4caa-84ab-0c0644741cf7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" Jul 08 02:59:25.352927 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.352906 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bh9f8\"" Jul 08 02:59:25.353333 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.353317 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 02:59:25.353435 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.353421 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 02:59:25.353491 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.353425 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 02:59:25.353535 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.353512 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a6d5cc2-87d6-4cc7-9711-349a152ff665-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.353712 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.353696 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 02:59:25.354175 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.354162 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 02:59:25.361514 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.361487 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj68v\" (UniqueName: \"kubernetes.io/projected/6aff4eef-9c66-4045-855c-08b914b03135-kube-api-access-vj68v\") pod \"managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb\" (UID: \"6aff4eef-9c66-4045-855c-08b914b03135\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb" Jul 08 02:59:25.361889 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.361871 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24nh9\" (UniqueName: \"kubernetes.io/projected/f4413f61-b3e4-4caa-84ab-0c0644741cf7-kube-api-access-24nh9\") pod \"klusterlet-addon-workmgr-84f4b8c98f-vwg4p\" (UID: \"f4413f61-b3e4-4caa-84ab-0c0644741cf7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" Jul 08 02:59:25.362013 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.361994 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-778j8\" (UniqueName: \"kubernetes.io/projected/7a6d5cc2-87d6-4cc7-9711-349a152ff665-kube-api-access-778j8\") pod \"cluster-proxy-proxy-agent-99975b4bc-jq5rk\" (UID: \"7a6d5cc2-87d6-4cc7-9711-349a152ff665\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.366525 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.366503 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8f8c64965-gjw9f"] Jul 08 02:59:25.440583 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.440556 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb" Jul 08 02:59:25.449462 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.449433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e78201e6-cbb1-4203-a1f4-103c40ceb783-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-64bj8\" (UID: \"e78201e6-cbb1-4203-a1f4-103c40ceb783\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-64bj8" Jul 08 02:59:25.449589 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.449470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-oauth-serving-cert\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.449589 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.449523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-config\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.449589 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.449560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k622k\" (UniqueName: \"kubernetes.io/projected/819aac06-7339-40a7-b54d-cffb09e62092-kube-api-access-k622k\") pod \"downloads-685bfb5f64-cgq8z\" (UID: \"819aac06-7339-40a7-b54d-cffb09e62092\") " pod="openshift-console/downloads-685bfb5f64-cgq8z" Jul 08 02:59:25.449589 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.449586 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-service-ca\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.449773 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.449721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" Jul 08 02:59:25.449773 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.449745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-serving-cert\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.449886 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.449778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-oauth-config\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.449886 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.449829 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4fg\" (UniqueName: \"kubernetes.io/projected/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-kube-api-access-lj4fg\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.452170 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.452148 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e78201e6-cbb1-4203-a1f4-103c40ceb783-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-64bj8\" (UID: \"e78201e6-cbb1-4203-a1f4-103c40ceb783\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-64bj8" Jul 08 02:59:25.459674 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.459651 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" Jul 08 02:59:25.461380 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.461360 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k622k\" (UniqueName: \"kubernetes.io/projected/819aac06-7339-40a7-b54d-cffb09e62092-kube-api-access-k622k\") pod \"downloads-685bfb5f64-cgq8z\" (UID: \"819aac06-7339-40a7-b54d-cffb09e62092\") " pod="openshift-console/downloads-685bfb5f64-cgq8z" Jul 08 02:59:25.521267 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.520987 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-cgq8z" Jul 08 02:59:25.533496 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.527951 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-64bj8" Jul 08 02:59:25.556192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.552705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-service-ca\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.556192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.552754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-serving-cert\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.556192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.552783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-oauth-config\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.556192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.552821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4fg\" (UniqueName: \"kubernetes.io/projected/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-kube-api-access-lj4fg\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.556192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.552889 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-oauth-serving-cert\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.556192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.552946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-config\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.556192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.554203 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-service-ca\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.556192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.553661 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-config\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.556629 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.556300 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-oauth-serving-cert\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.559771 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.558634 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-serving-cert\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.561158 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.560898 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-oauth-config\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.564812 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.564770 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4fg\" (UniqueName: \"kubernetes.io/projected/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-kube-api-access-lj4fg\") pod \"console-8f8c64965-gjw9f\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.610807 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.610726 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb"] Jul 08 02:59:25.624536 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:25.622710 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6aff4eef_9c66_4045_855c_08b914b03135.slice/crio-f6cd4de129939ddd0a2b21e6bd37ad346689514be55b8077abf8bd7b30e2241d WatchSource:0}: Error finding container f6cd4de129939ddd0a2b21e6bd37ad346689514be55b8077abf8bd7b30e2241d: Status 404 returned error can't find the container with id f6cd4de129939ddd0a2b21e6bd37ad346689514be55b8077abf8bd7b30e2241d Jul 08 02:59:25.637330 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.636404 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p"] Jul 08 02:59:25.640940 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:25.640891 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4413f61_b3e4_4caa_84ab_0c0644741cf7.slice/crio-cc29fdaf05d4c40e4c3bb8984021db11b347e66482e56d6bcfd5a0521905d851 WatchSource:0}: Error finding container cc29fdaf05d4c40e4c3bb8984021db11b347e66482e56d6bcfd5a0521905d851: Status 404 returned error can't find the container with id cc29fdaf05d4c40e4c3bb8984021db11b347e66482e56d6bcfd5a0521905d851 Jul 08 02:59:25.657306 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.657252 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk"] Jul 08 02:59:25.661200 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:25.661162 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a6d5cc2_87d6_4cc7_9711_349a152ff665.slice/crio-d73e273204c22eb3856b732158b68dd18dba97e31ee38dee09ae601c4dc6d505 WatchSource:0}: Error finding container d73e273204c22eb3856b732158b68dd18dba97e31ee38dee09ae601c4dc6d505: Status 404 returned error can't find the container with id d73e273204c22eb3856b732158b68dd18dba97e31ee38dee09ae601c4dc6d505 Jul 08 02:59:25.677748 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.677723 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:25.713180 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.713105 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-64bj8"] Jul 08 02:59:25.715925 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:25.715879 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78201e6_cbb1_4203_a1f4_103c40ceb783.slice/crio-82d458a22b2a46d4e004ccb77cefcd63eb37d3abbf4b80c2fba554f924a481ab WatchSource:0}: Error finding container 82d458a22b2a46d4e004ccb77cefcd63eb37d3abbf4b80c2fba554f924a481ab: Status 404 returned error can't find the container with id 82d458a22b2a46d4e004ccb77cefcd63eb37d3abbf4b80c2fba554f924a481ab Jul 08 02:59:25.732885 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.732862 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-cgq8z"] Jul 08 02:59:25.736747 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:25.736127 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819aac06_7339_40a7_b54d_cffb09e62092.slice/crio-ec2a0eecdfefcabbb3c804abceafe97fe9364370e73c670ae97fa89091569a5c WatchSource:0}: Error finding container ec2a0eecdfefcabbb3c804abceafe97fe9364370e73c670ae97fa89091569a5c: Status 404 returned error can't find the container with id ec2a0eecdfefcabbb3c804abceafe97fe9364370e73c670ae97fa89091569a5c Jul 08 02:59:25.817421 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:25.817392 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8f8c64965-gjw9f"] Jul 08 02:59:25.820415 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:25.820378 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6f8fe0_46b4_40bf_9455_dff1f85c8e76.slice/crio-9508219c1aa5dd3cc838acff774316daa05d41010f37ecdc98c48d64b5d4db49 WatchSource:0}: Error finding container 9508219c1aa5dd3cc838acff774316daa05d41010f37ecdc98c48d64b5d4db49: Status 404 returned error can't find the container with id 9508219c1aa5dd3cc838acff774316daa05d41010f37ecdc98c48d64b5d4db49 Jul 08 02:59:26.065768 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:26.065728 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb" event={"ID":"6aff4eef-9c66-4045-855c-08b914b03135","Type":"ContainerStarted","Data":"f6cd4de129939ddd0a2b21e6bd37ad346689514be55b8077abf8bd7b30e2241d"} Jul 08 02:59:26.067347 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:26.067312 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" event={"ID":"f4413f61-b3e4-4caa-84ab-0c0644741cf7","Type":"ContainerStarted","Data":"cc29fdaf05d4c40e4c3bb8984021db11b347e66482e56d6bcfd5a0521905d851"} Jul 08 02:59:26.072391 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:26.072359 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" event={"ID":"7a6d5cc2-87d6-4cc7-9711-349a152ff665","Type":"ContainerStarted","Data":"d73e273204c22eb3856b732158b68dd18dba97e31ee38dee09ae601c4dc6d505"} Jul 08 02:59:26.073827 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:26.073802 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-cgq8z" event={"ID":"819aac06-7339-40a7-b54d-cffb09e62092","Type":"ContainerStarted","Data":"ec2a0eecdfefcabbb3c804abceafe97fe9364370e73c670ae97fa89091569a5c"} Jul 08 02:59:26.075959 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:26.075876 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-64bj8" event={"ID":"e78201e6-cbb1-4203-a1f4-103c40ceb783","Type":"ContainerStarted","Data":"82d458a22b2a46d4e004ccb77cefcd63eb37d3abbf4b80c2fba554f924a481ab"} Jul 08 02:59:26.077434 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:26.077409 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f8c64965-gjw9f" event={"ID":"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76","Type":"ContainerStarted","Data":"9508219c1aa5dd3cc838acff774316daa05d41010f37ecdc98c48d64b5d4db49"} Jul 08 02:59:26.912690 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:26.911235 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d649cd899-xxsrz"] Jul 08 02:59:26.915701 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:26.915100 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:26.931407 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:26.931378 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 08 02:59:26.935119 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:26.935090 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d649cd899-xxsrz"] Jul 08 02:59:27.074473 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.074410 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93a506de-ab52-48e5-851e-647d7624754c-console-oauth-config\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.074658 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.074478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93a506de-ab52-48e5-851e-647d7624754c-console-serving-cert\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.074658 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.074517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-service-ca\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.074658 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.074575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-trusted-ca-bundle\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.074658 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.074629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-console-config\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.074658 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.074652 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-oauth-serving-cert\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.074931 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.074688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh4t8\" (UniqueName: \"kubernetes.io/projected/93a506de-ab52-48e5-851e-647d7624754c-kube-api-access-xh4t8\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.175767 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.175578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-trusted-ca-bundle\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.175767 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.175665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-console-config\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.175767 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.175690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-oauth-serving-cert\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.175767 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.175768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh4t8\" (UniqueName: \"kubernetes.io/projected/93a506de-ab52-48e5-851e-647d7624754c-kube-api-access-xh4t8\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.176347 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.175844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93a506de-ab52-48e5-851e-647d7624754c-console-oauth-config\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.176347 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.175876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93a506de-ab52-48e5-851e-647d7624754c-console-serving-cert\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.176347 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.175907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-service-ca\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.176640 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.176565 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-trusted-ca-bundle\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.176701 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.176684 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-service-ca\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.177301 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.176946 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-console-config\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.177301 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.177145 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-oauth-serving-cert\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.188748 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.188701 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93a506de-ab52-48e5-851e-647d7624754c-console-serving-cert\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.190429 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.190375 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93a506de-ab52-48e5-851e-647d7624754c-console-oauth-config\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.198987 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.198938 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh4t8\" (UniqueName: \"kubernetes.io/projected/93a506de-ab52-48e5-851e-647d7624754c-kube-api-access-xh4t8\") pod \"console-7d649cd899-xxsrz\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.237077 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.236898 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:27.469389 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:27.469268 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d649cd899-xxsrz"] Jul 08 02:59:28.026016 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:28.025888 2576 patch_prober.go:28] interesting pod/image-registry-6c4cc5dff5-lx8g9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 08 02:59:28.026016 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:28.025951 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" podUID="80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 02:59:28.408942 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:28.404714 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a506de_ab52_48e5_851e_647d7624754c.slice/crio-249cf7354abfb2c704128d9e7b04885d0e3870d2a0dbc7b7d8dc69c8907ab575 WatchSource:0}: Error finding container 249cf7354abfb2c704128d9e7b04885d0e3870d2a0dbc7b7d8dc69c8907ab575: Status 404 returned error can't find the container with id 249cf7354abfb2c704128d9e7b04885d0e3870d2a0dbc7b7d8dc69c8907ab575 Jul 08 02:59:29.106840 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:29.106799 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d649cd899-xxsrz" event={"ID":"93a506de-ab52-48e5-851e-647d7624754c","Type":"ContainerStarted","Data":"249cf7354abfb2c704128d9e7b04885d0e3870d2a0dbc7b7d8dc69c8907ab575"} Jul 08 02:59:29.969013 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:29.968987 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 02:59:35.141789 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.141746 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d649cd899-xxsrz" event={"ID":"93a506de-ab52-48e5-851e-647d7624754c","Type":"ContainerStarted","Data":"cf00f67e1fbd1668277c709c5a81db76703eae38fb812ccb5923a7418cd23d66"} Jul 08 02:59:35.144416 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.144235 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-64bj8" event={"ID":"e78201e6-cbb1-4203-a1f4-103c40ceb783","Type":"ContainerStarted","Data":"ad377a1a43ab813d05407c0bf86efde579cd5d050d07825aee4b332a705fd171"} Jul 08 02:59:35.144747 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.144732 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-64bj8" Jul 08 02:59:35.147969 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.147947 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f8c64965-gjw9f" event={"ID":"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76","Type":"ContainerStarted","Data":"b069c341078c107f7cab34cf58fd7cceb9d68c83cace9ee31d39a792483504c7"} Jul 08 02:59:35.151952 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.151912 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-64bj8" Jul 08 02:59:35.153127 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.153102 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb" event={"ID":"6aff4eef-9c66-4045-855c-08b914b03135","Type":"ContainerStarted","Data":"730b3c8e4792a2e372952736e62257c462417b07e44781b29bc7465b5c8c7fae"} Jul 08 02:59:35.155598 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.155433 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" event={"ID":"f4413f61-b3e4-4caa-84ab-0c0644741cf7","Type":"ContainerStarted","Data":"e75b9822e3d896c2b66c15eb6149c489a9d2b4d6ad04eae37639f38e5bf59725"} Jul 08 02:59:35.155598 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.155581 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" Jul 08 02:59:35.157804 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.157771 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" Jul 08 02:59:35.159395 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.159376 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" event={"ID":"7a6d5cc2-87d6-4cc7-9711-349a152ff665","Type":"ContainerStarted","Data":"52d2a85b655135a46950480b8c44f820770c15c6fb951084ed91d780ffb21206"} Jul 08 02:59:35.159553 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.159543 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" event={"ID":"7a6d5cc2-87d6-4cc7-9711-349a152ff665","Type":"ContainerStarted","Data":"264b5e961acaf626ce364a93f0609d45ed09676a15f8ad3122e5a6c77880be60"} Jul 08 02:59:35.161977 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.161141 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d649cd899-xxsrz" podStartSLOduration=2.691773392 podStartE2EDuration="9.161128256s" totalImagesPullingTime="6.469354864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:35.160445082 +0000 UTC m=+94.150893120" watchObservedRunningTime="2026-07-08 02:59:35.161128256 +0000 UTC m=+94.151576308" Jul 08 02:59:35.176388 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.176340 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-64bj8" podStartSLOduration=1.072183847 podStartE2EDuration="10.176323376s" totalImagesPullingTime="9.104139529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:35.175976142 +0000 UTC m=+94.166424182" watchObservedRunningTime="2026-07-08 02:59:35.176323376 +0000 UTC m=+94.166771414" Jul 08 02:59:35.192603 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.192390 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6cfb456d-vn9nb" podStartSLOduration=0.991834255 podStartE2EDuration="10.192372346s" totalImagesPullingTime="9.200538091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:35.191383415 +0000 UTC m=+94.181831455" watchObservedRunningTime="2026-07-08 02:59:35.192372346 +0000 UTC m=+94.182820384" Jul 08 02:59:35.216134 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.216076 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f4b8c98f-vwg4p" podStartSLOduration=0.987284543 podStartE2EDuration="10.21602285s" totalImagesPullingTime="9.228738307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:35.215518523 +0000 UTC m=+94.205966561" watchObservedRunningTime="2026-07-08 02:59:35.21602285 +0000 UTC m=+94.206470887" Jul 08 02:59:35.236144 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.236091 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8f8c64965-gjw9f" podStartSLOduration=1.233790486 podStartE2EDuration="10.236068517s" totalImagesPullingTime="9.002278031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:35.23409648 +0000 UTC m=+94.224544519" watchObservedRunningTime="2026-07-08 02:59:35.236068517 +0000 UTC m=+94.226516556" Jul 08 02:59:35.678801 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.678756 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:35.678978 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.678845 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:35.685001 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:35.684977 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:36.172014 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:36.171964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" event={"ID":"7a6d5cc2-87d6-4cc7-9711-349a152ff665","Type":"ContainerStarted","Data":"1ff7147d00a670313582129bf69df467ed4f6964e103c58ecc3d3a0d6c70b711"} Jul 08 02:59:36.178335 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:36.178304 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 02:59:36.219544 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:36.219495 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-99975b4bc-jq5rk" podStartSLOduration=2.01051157 podStartE2EDuration="11.219472396s" totalImagesPullingTime="9.208960826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:36.197922123 +0000 UTC m=+95.188370161" watchObservedRunningTime="2026-07-08 02:59:36.219472396 +0000 UTC m=+95.209920432" Jul 08 02:59:37.238449 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:37.238415 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:37.239135 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:37.238483 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:37.244378 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:37.244354 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:37.956636 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:37.956571 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bwz4j" Jul 08 02:59:38.187805 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:38.187774 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 02:59:38.238855 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:38.238737 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8f8c64965-gjw9f"] Jul 08 02:59:42.708668 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.708631 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s5h8n"] Jul 08 02:59:42.873007 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.872969 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:42.877309 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.877288 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9rm8r\"" Jul 08 02:59:42.877498 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.877482 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 02:59:42.877661 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.877648 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 02:59:42.877842 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.877827 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 02:59:42.877985 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.877973 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 02:59:42.927188 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.927151 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7ae4070-9290-4092-bc23-7e1ce413daff-root\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:42.927368 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.927199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmfb8\" (UniqueName: \"kubernetes.io/projected/f7ae4070-9290-4092-bc23-7e1ce413daff-kube-api-access-lmfb8\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:42.927368 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.927259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-tls\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:42.927368 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.927285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-wtmp\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:42.927368 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.927310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7ae4070-9290-4092-bc23-7e1ce413daff-metrics-client-ca\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:42.927561 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.927415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7ae4070-9290-4092-bc23-7e1ce413daff-sys\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:42.927561 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.927466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:42.927659 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.927558 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-textfile\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:42.927659 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:42.927584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.028867 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.028779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7ae4070-9290-4092-bc23-7e1ce413daff-root\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.028867 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.028818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmfb8\" (UniqueName: \"kubernetes.io/projected/f7ae4070-9290-4092-bc23-7e1ce413daff-kube-api-access-lmfb8\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.028867 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.028868 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-tls\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.029175 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.028902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-wtmp\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.029175 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.028904 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7ae4070-9290-4092-bc23-7e1ce413daff-root\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.029175 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.029017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7ae4070-9290-4092-bc23-7e1ce413daff-metrics-client-ca\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.029307 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.029220 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-wtmp\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.029307 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.029264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7ae4070-9290-4092-bc23-7e1ce413daff-sys\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.029411 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.029305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.029411 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.029362 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7ae4070-9290-4092-bc23-7e1ce413daff-sys\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.029411 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.029364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-textfile\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.029551 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.029422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.029620 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.029595 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7ae4070-9290-4092-bc23-7e1ce413daff-metrics-client-ca\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.029676 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.029644 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-textfile\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.029983 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.029956 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.031697 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.031674 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-tls\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.032147 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.032125 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7ae4070-9290-4092-bc23-7e1ce413daff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.039521 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.039495 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmfb8\" (UniqueName: \"kubernetes.io/projected/f7ae4070-9290-4092-bc23-7e1ce413daff-kube-api-access-lmfb8\") pod \"node-exporter-s5h8n\" (UID: \"f7ae4070-9290-4092-bc23-7e1ce413daff\") " pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:43.185637 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:43.185602 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s5h8n" Jul 08 02:59:46.004386 ip-10-0-142-99 kubenswrapper[2576]: W0708 02:59:46.004353 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ae4070_9290_4092_bc23_7e1ce413daff.slice/crio-d8af479e2747d1ed4d1b599ae2387b4d2a4733f1855b5f438a3eb0729ce6ed6d WatchSource:0}: Error finding container d8af479e2747d1ed4d1b599ae2387b4d2a4733f1855b5f438a3eb0729ce6ed6d: Status 404 returned error can't find the container with id d8af479e2747d1ed4d1b599ae2387b4d2a4733f1855b5f438a3eb0729ce6ed6d Jul 08 02:59:46.226030 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:46.225986 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5h8n" event={"ID":"f7ae4070-9290-4092-bc23-7e1ce413daff","Type":"ContainerStarted","Data":"d8af479e2747d1ed4d1b599ae2387b4d2a4733f1855b5f438a3eb0729ce6ed6d"} Jul 08 02:59:47.233192 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:47.233152 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-cgq8z" event={"ID":"819aac06-7339-40a7-b54d-cffb09e62092","Type":"ContainerStarted","Data":"33b81ad1a581c7e4ced28253246a21f9c14492af8cbc91237dd763f94f1c8e4e"} Jul 08 02:59:47.233682 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:47.233392 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-cgq8z" Jul 08 02:59:47.244574 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:47.244547 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-cgq8z" Jul 08 02:59:47.253834 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:47.253781 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-cgq8z" podStartSLOduration=1.566703776 podStartE2EDuration="22.253764204s" totalImagesPullingTime="20.687060428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:47.25219776 +0000 UTC m=+106.242645809" watchObservedRunningTime="2026-07-08 02:59:47.253764204 +0000 UTC m=+106.244212243" Jul 08 02:59:48.241014 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:48.240976 2576 generic.go:356] "Generic (PLEG): container finished" podID="f7ae4070-9290-4092-bc23-7e1ce413daff" containerID="abfafeff840b5021cb38ccfba5b21f613985ae3640e5411730824803840c245a" exitCode=0 Jul 08 02:59:48.241504 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:48.241087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5h8n" event={"ID":"f7ae4070-9290-4092-bc23-7e1ce413daff","Type":"ContainerDied","Data":"abfafeff840b5021cb38ccfba5b21f613985ae3640e5411730824803840c245a"} Jul 08 02:59:49.249199 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:49.249159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5h8n" event={"ID":"f7ae4070-9290-4092-bc23-7e1ce413daff","Type":"ContainerStarted","Data":"d64d0c53c8dd3adfed2d821a4b38960c13b79fa45cb8373c5fb88561f41c5281"} Jul 08 02:59:49.249740 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:49.249231 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5h8n" event={"ID":"f7ae4070-9290-4092-bc23-7e1ce413daff","Type":"ContainerStarted","Data":"a1176d70146d4cdc59fdb8ca627e978fe3f7f3868dba9b384972cbbfb38d7d2d"} Jul 08 02:59:49.272468 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:49.272420 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s5h8n" podStartSLOduration=5.671236396 podStartE2EDuration="7.272405128s" totalImagesPullingTime="1.480952423s" totalInitContainerRuntime="120.216309ms" isStatefulPod=true podCreationTimestamp="2026-07-08 02:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:49.269768233 +0000 UTC m=+108.260216309" watchObservedRunningTime="2026-07-08 02:59:49.272405128 +0000 UTC m=+108.262853165" Jul 08 02:59:54.468720 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:54.468685 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c4cc5dff5-lx8g9"] Jul 08 02:59:58.609052 ip-10-0-142-99 kubenswrapper[2576]: I0708 02:59:58.609004 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d649cd899-xxsrz"] Jul 08 03:00:04.211058 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.211000 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-8f8c64965-gjw9f" podUID="ba6f8fe0-46b4-40bf-9455-dff1f85c8e76" containerName="console" containerID="cri-o://b069c341078c107f7cab34cf58fd7cceb9d68c83cace9ee31d39a792483504c7" gracePeriod=15 Jul 08 03:00:04.490477 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.490418 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 03:00:04.638179 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.638148 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj4fg\" (UniqueName: \"kubernetes.io/projected/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-kube-api-access-lj4fg\") pod \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " Jul 08 03:00:04.638362 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.638193 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-serving-cert\") pod \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " Jul 08 03:00:04.638362 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.638228 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-service-ca\") pod \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " Jul 08 03:00:04.638362 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.638266 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-config\") pod \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " Jul 08 03:00:04.638362 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.638302 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-oauth-serving-cert\") pod \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " Jul 08 03:00:04.638575 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.638376 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-oauth-config\") pod \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\" (UID: \"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76\") " Jul 08 03:00:04.638712 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.638686 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-service-ca" (OuterVolumeSpecName: "service-ca") pod "ba6f8fe0-46b4-40bf-9455-dff1f85c8e76" (UID: "ba6f8fe0-46b4-40bf-9455-dff1f85c8e76"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:00:04.638941 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.638705 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-config" (OuterVolumeSpecName: "console-config") pod "ba6f8fe0-46b4-40bf-9455-dff1f85c8e76" (UID: "ba6f8fe0-46b4-40bf-9455-dff1f85c8e76"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:00:04.638941 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.638790 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ba6f8fe0-46b4-40bf-9455-dff1f85c8e76" (UID: "ba6f8fe0-46b4-40bf-9455-dff1f85c8e76"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:00:04.640670 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.640646 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ba6f8fe0-46b4-40bf-9455-dff1f85c8e76" (UID: "ba6f8fe0-46b4-40bf-9455-dff1f85c8e76"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:00:04.640750 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.640699 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-kube-api-access-lj4fg" (OuterVolumeSpecName: "kube-api-access-lj4fg") pod "ba6f8fe0-46b4-40bf-9455-dff1f85c8e76" (UID: "ba6f8fe0-46b4-40bf-9455-dff1f85c8e76"). InnerVolumeSpecName "kube-api-access-lj4fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:00:04.640750 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.640706 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ba6f8fe0-46b4-40bf-9455-dff1f85c8e76" (UID: "ba6f8fe0-46b4-40bf-9455-dff1f85c8e76"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:00:04.740010 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.739979 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-serving-cert\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:04.740010 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.740007 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-service-ca\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:04.740010 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.740017 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-config\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:04.740229 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.740024 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-oauth-serving-cert\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:04.740229 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.740033 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-console-oauth-config\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:04.740229 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:04.740063 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj4fg\" (UniqueName: \"kubernetes.io/projected/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76-kube-api-access-lj4fg\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:05.320987 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:05.320955 2576 generic.go:356] "Generic (PLEG): container finished" podID="ba6f8fe0-46b4-40bf-9455-dff1f85c8e76" containerID="b069c341078c107f7cab34cf58fd7cceb9d68c83cace9ee31d39a792483504c7" exitCode=0 Jul 08 03:00:05.321459 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:05.321006 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f8c64965-gjw9f" Jul 08 03:00:05.321459 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:05.321061 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f8c64965-gjw9f" event={"ID":"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76","Type":"ContainerDied","Data":"b069c341078c107f7cab34cf58fd7cceb9d68c83cace9ee31d39a792483504c7"} Jul 08 03:00:05.321459 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:05.321123 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f8c64965-gjw9f" event={"ID":"ba6f8fe0-46b4-40bf-9455-dff1f85c8e76","Type":"ContainerDied","Data":"9508219c1aa5dd3cc838acff774316daa05d41010f37ecdc98c48d64b5d4db49"} Jul 08 03:00:05.321459 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:05.321165 2576 scope.go:117] "RemoveContainer" containerID="b069c341078c107f7cab34cf58fd7cceb9d68c83cace9ee31d39a792483504c7" Jul 08 03:00:05.330148 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:05.330029 2576 scope.go:117] "RemoveContainer" containerID="b069c341078c107f7cab34cf58fd7cceb9d68c83cace9ee31d39a792483504c7" Jul 08 03:00:05.332847 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:00:05.332768 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b069c341078c107f7cab34cf58fd7cceb9d68c83cace9ee31d39a792483504c7\": container with ID starting with b069c341078c107f7cab34cf58fd7cceb9d68c83cace9ee31d39a792483504c7 not found: ID does not exist" containerID="b069c341078c107f7cab34cf58fd7cceb9d68c83cace9ee31d39a792483504c7" Jul 08 03:00:05.332847 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:05.332804 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b069c341078c107f7cab34cf58fd7cceb9d68c83cace9ee31d39a792483504c7"} err="failed to get container status \"b069c341078c107f7cab34cf58fd7cceb9d68c83cace9ee31d39a792483504c7\": rpc error: code = NotFound desc = could not find container \"b069c341078c107f7cab34cf58fd7cceb9d68c83cace9ee31d39a792483504c7\": container with ID starting with b069c341078c107f7cab34cf58fd7cceb9d68c83cace9ee31d39a792483504c7 not found: ID does not exist" Jul 08 03:00:05.342596 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:05.342576 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8f8c64965-gjw9f"] Jul 08 03:00:05.346575 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:05.346556 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8f8c64965-gjw9f"] Jul 08 03:00:05.525164 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:05.525128 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba6f8fe0-46b4-40bf-9455-dff1f85c8e76" path="/var/lib/kubelet/pods/ba6f8fe0-46b4-40bf-9455-dff1f85c8e76/volumes" Jul 08 03:00:10.283112 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:10.283069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs\") pod \"network-metrics-daemon-wdvzz\" (UID: \"c392372d-4d45-4636-8058-57fd87930e8e\") " pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 03:00:10.285431 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:10.285406 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c392372d-4d45-4636-8058-57fd87930e8e-metrics-certs\") pod \"network-metrics-daemon-wdvzz\" (UID: \"c392372d-4d45-4636-8058-57fd87930e8e\") " pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 03:00:10.547015 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:10.546941 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fsx5r\"" Jul 08 03:00:10.554985 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:10.554956 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdvzz" Jul 08 03:00:10.717769 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:10.717651 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wdvzz"] Jul 08 03:00:10.720420 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:00:10.720395 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc392372d_4d45_4636_8058_57fd87930e8e.slice/crio-56437a929995e7897f8cbddcab7cd726b9d089a9611d881eb98dc8633a5e7a30 WatchSource:0}: Error finding container 56437a929995e7897f8cbddcab7cd726b9d089a9611d881eb98dc8633a5e7a30: Status 404 returned error can't find the container with id 56437a929995e7897f8cbddcab7cd726b9d089a9611d881eb98dc8633a5e7a30 Jul 08 03:00:11.351113 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:11.351009 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wdvzz" event={"ID":"c392372d-4d45-4636-8058-57fd87930e8e","Type":"ContainerStarted","Data":"56437a929995e7897f8cbddcab7cd726b9d089a9611d881eb98dc8633a5e7a30"} Jul 08 03:00:13.360795 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:13.360749 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wdvzz" event={"ID":"c392372d-4d45-4636-8058-57fd87930e8e","Type":"ContainerStarted","Data":"a9f7ad8ae15adb9b45aa57a64af4f6f3a0e702fe5f7d298fab63c5f52cc4ca0c"} Jul 08 03:00:13.361346 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:13.360865 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wdvzz" event={"ID":"c392372d-4d45-4636-8058-57fd87930e8e","Type":"ContainerStarted","Data":"e913d3722ae8518c7c9826f0a6d0b2f8d0fa32347a2f01666850df4a4c4c8b29"} Jul 08 03:00:13.362222 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:13.362198 2576 generic.go:356] "Generic (PLEG): container finished" podID="685a7232-5ff7-4f67-b949-1c41b98bb3c3" containerID="28d45af4ea3605a70cb6732d686594ee43e2c6c5c0c7c1e09e603b98c94bbbc1" exitCode=0 Jul 08 03:00:13.362335 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:13.362235 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" event={"ID":"685a7232-5ff7-4f67-b949-1c41b98bb3c3","Type":"ContainerDied","Data":"28d45af4ea3605a70cb6732d686594ee43e2c6c5c0c7c1e09e603b98c94bbbc1"} Jul 08 03:00:13.362523 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:13.362509 2576 scope.go:117] "RemoveContainer" containerID="28d45af4ea3605a70cb6732d686594ee43e2c6c5c0c7c1e09e603b98c94bbbc1" Jul 08 03:00:13.378516 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:13.378463 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wdvzz" podStartSLOduration=129.721206915 podStartE2EDuration="2m11.378448027s" totalImagesPullingTime="1.657241112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:00:13.376777907 +0000 UTC m=+132.367225944" watchObservedRunningTime="2026-07-08 03:00:13.378448027 +0000 UTC m=+132.368896076" Jul 08 03:00:14.368492 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:14.368454 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-45htx" event={"ID":"685a7232-5ff7-4f67-b949-1c41b98bb3c3","Type":"ContainerStarted","Data":"5879ad747695a1bdb69ae390d99ca3d71bed889b1a722434c666efa334fffe28"} Jul 08 03:00:19.494558 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.494490 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" podUID="80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" containerName="registry" containerID="cri-o://e068cb661c14c5bfcb9da2bc2e364e04e7ebd217a1b5f5112c8b0d7508a4ea0c" gracePeriod=30 Jul 08 03:00:19.774001 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.773935 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 03:00:19.859068 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.859019 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-bound-sa-token\") pod \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " Jul 08 03:00:19.859231 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.859077 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-ca-trust-extracted\") pod \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " Jul 08 03:00:19.859231 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.859203 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-certificates\") pod \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " Jul 08 03:00:19.859330 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.859236 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-image-registry-private-configuration\") pod \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " Jul 08 03:00:19.859330 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.859269 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-installation-pull-secrets\") pod \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " Jul 08 03:00:19.859431 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.859358 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls\") pod \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " Jul 08 03:00:19.859431 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.859421 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-trusted-ca\") pod \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " Jul 08 03:00:19.859534 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.859479 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdn9k\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-kube-api-access-qdn9k\") pod \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\" (UID: \"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5\") " Jul 08 03:00:19.859662 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.859634 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" (UID: "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:00:19.859795 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.859780 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-certificates\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:19.860065 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.860016 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" (UID: "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:00:19.862105 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.862076 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" (UID: "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:00:19.862205 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.862169 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" (UID: "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:00:19.862271 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.862237 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-kube-api-access-qdn9k" (OuterVolumeSpecName: "kube-api-access-qdn9k") pod "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" (UID: "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5"). InnerVolumeSpecName "kube-api-access-qdn9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:00:19.862271 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.862253 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" (UID: "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:00:19.862353 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.862299 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" (UID: "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:00:19.868016 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.867991 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" (UID: "80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:00:19.960313 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.960279 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-bound-sa-token\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:19.960313 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.960306 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-ca-trust-extracted\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:19.960313 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.960316 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-image-registry-private-configuration\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:19.960538 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.960325 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-installation-pull-secrets\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:19.960538 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.960335 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-registry-tls\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:19.960538 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.960343 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-trusted-ca\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:19.960538 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:19.960351 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdn9k\" (UniqueName: \"kubernetes.io/projected/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5-kube-api-access-qdn9k\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:20.397491 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:20.397455 2576 generic.go:356] "Generic (PLEG): container finished" podID="80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" containerID="e068cb661c14c5bfcb9da2bc2e364e04e7ebd217a1b5f5112c8b0d7508a4ea0c" exitCode=0 Jul 08 03:00:20.397634 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:20.397500 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" Jul 08 03:00:20.397634 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:20.397545 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" event={"ID":"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5","Type":"ContainerDied","Data":"e068cb661c14c5bfcb9da2bc2e364e04e7ebd217a1b5f5112c8b0d7508a4ea0c"} Jul 08 03:00:20.397634 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:20.397594 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c4cc5dff5-lx8g9" event={"ID":"80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5","Type":"ContainerDied","Data":"21507af95ea6410ec283a10dd796d90c7d2c882cefdba6afccecbed9987dc129"} Jul 08 03:00:20.397634 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:20.397616 2576 scope.go:117] "RemoveContainer" containerID="e068cb661c14c5bfcb9da2bc2e364e04e7ebd217a1b5f5112c8b0d7508a4ea0c" Jul 08 03:00:20.406972 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:20.406955 2576 scope.go:117] "RemoveContainer" containerID="e068cb661c14c5bfcb9da2bc2e364e04e7ebd217a1b5f5112c8b0d7508a4ea0c" Jul 08 03:00:20.407267 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:00:20.407246 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e068cb661c14c5bfcb9da2bc2e364e04e7ebd217a1b5f5112c8b0d7508a4ea0c\": container with ID starting with e068cb661c14c5bfcb9da2bc2e364e04e7ebd217a1b5f5112c8b0d7508a4ea0c not found: ID does not exist" containerID="e068cb661c14c5bfcb9da2bc2e364e04e7ebd217a1b5f5112c8b0d7508a4ea0c" Jul 08 03:00:20.407337 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:20.407280 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e068cb661c14c5bfcb9da2bc2e364e04e7ebd217a1b5f5112c8b0d7508a4ea0c"} err="failed to get container status \"e068cb661c14c5bfcb9da2bc2e364e04e7ebd217a1b5f5112c8b0d7508a4ea0c\": rpc error: code = NotFound desc = could not find container \"e068cb661c14c5bfcb9da2bc2e364e04e7ebd217a1b5f5112c8b0d7508a4ea0c\": container with ID starting with e068cb661c14c5bfcb9da2bc2e364e04e7ebd217a1b5f5112c8b0d7508a4ea0c not found: ID does not exist" Jul 08 03:00:20.420093 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:20.420065 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c4cc5dff5-lx8g9"] Jul 08 03:00:20.423837 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:20.423817 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c4cc5dff5-lx8g9"] Jul 08 03:00:21.525315 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:21.525280 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" path="/var/lib/kubelet/pods/80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5/volumes" Jul 08 03:00:23.630949 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.630876 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7d649cd899-xxsrz" podUID="93a506de-ab52-48e5-851e-647d7624754c" containerName="console" containerID="cri-o://cf00f67e1fbd1668277c709c5a81db76703eae38fb812ccb5923a7418cd23d66" gracePeriod=15 Jul 08 03:00:23.913627 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.913593 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 03:00:23.989592 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.989563 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-service-ca\") pod \"93a506de-ab52-48e5-851e-647d7624754c\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " Jul 08 03:00:23.989750 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.989609 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93a506de-ab52-48e5-851e-647d7624754c-console-oauth-config\") pod \"93a506de-ab52-48e5-851e-647d7624754c\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " Jul 08 03:00:23.989750 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.989646 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-oauth-serving-cert\") pod \"93a506de-ab52-48e5-851e-647d7624754c\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " Jul 08 03:00:23.989750 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.989680 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-trusted-ca-bundle\") pod \"93a506de-ab52-48e5-851e-647d7624754c\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " Jul 08 03:00:23.989906 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.989754 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh4t8\" (UniqueName: \"kubernetes.io/projected/93a506de-ab52-48e5-851e-647d7624754c-kube-api-access-xh4t8\") pod \"93a506de-ab52-48e5-851e-647d7624754c\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " Jul 08 03:00:23.989906 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.989786 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-console-config\") pod \"93a506de-ab52-48e5-851e-647d7624754c\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " Jul 08 03:00:23.989906 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.989822 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93a506de-ab52-48e5-851e-647d7624754c-console-serving-cert\") pod \"93a506de-ab52-48e5-851e-647d7624754c\" (UID: \"93a506de-ab52-48e5-851e-647d7624754c\") " Jul 08 03:00:23.990086 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.989893 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-service-ca" (OuterVolumeSpecName: "service-ca") pod "93a506de-ab52-48e5-851e-647d7624754c" (UID: "93a506de-ab52-48e5-851e-647d7624754c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:00:23.990155 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.990093 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-service-ca\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:23.990155 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.990091 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "93a506de-ab52-48e5-851e-647d7624754c" (UID: "93a506de-ab52-48e5-851e-647d7624754c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:00:23.990155 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.990112 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "93a506de-ab52-48e5-851e-647d7624754c" (UID: "93a506de-ab52-48e5-851e-647d7624754c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:00:23.990287 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.990195 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-console-config" (OuterVolumeSpecName: "console-config") pod "93a506de-ab52-48e5-851e-647d7624754c" (UID: "93a506de-ab52-48e5-851e-647d7624754c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:00:23.991945 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.991921 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a506de-ab52-48e5-851e-647d7624754c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "93a506de-ab52-48e5-851e-647d7624754c" (UID: "93a506de-ab52-48e5-851e-647d7624754c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:00:23.992033 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.991950 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a506de-ab52-48e5-851e-647d7624754c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "93a506de-ab52-48e5-851e-647d7624754c" (UID: "93a506de-ab52-48e5-851e-647d7624754c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:00:23.992105 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:23.992083 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a506de-ab52-48e5-851e-647d7624754c-kube-api-access-xh4t8" (OuterVolumeSpecName: "kube-api-access-xh4t8") pod "93a506de-ab52-48e5-851e-647d7624754c" (UID: "93a506de-ab52-48e5-851e-647d7624754c"). InnerVolumeSpecName "kube-api-access-xh4t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:00:24.090628 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.090596 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh4t8\" (UniqueName: \"kubernetes.io/projected/93a506de-ab52-48e5-851e-647d7624754c-kube-api-access-xh4t8\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:24.090628 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.090623 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-console-config\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:24.090628 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.090633 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93a506de-ab52-48e5-851e-647d7624754c-console-serving-cert\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:24.090830 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.090641 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93a506de-ab52-48e5-851e-647d7624754c-console-oauth-config\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:24.090830 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.090650 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-oauth-serving-cert\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:24.090830 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.090658 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a506de-ab52-48e5-851e-647d7624754c-trusted-ca-bundle\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:00:24.418290 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.418251 2576 generic.go:356] "Generic (PLEG): container finished" podID="93a506de-ab52-48e5-851e-647d7624754c" containerID="cf00f67e1fbd1668277c709c5a81db76703eae38fb812ccb5923a7418cd23d66" exitCode=0 Jul 08 03:00:24.418290 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.418296 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d649cd899-xxsrz" Jul 08 03:00:24.418596 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.418343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d649cd899-xxsrz" event={"ID":"93a506de-ab52-48e5-851e-647d7624754c","Type":"ContainerDied","Data":"cf00f67e1fbd1668277c709c5a81db76703eae38fb812ccb5923a7418cd23d66"} Jul 08 03:00:24.418596 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.418408 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d649cd899-xxsrz" event={"ID":"93a506de-ab52-48e5-851e-647d7624754c","Type":"ContainerDied","Data":"249cf7354abfb2c704128d9e7b04885d0e3870d2a0dbc7b7d8dc69c8907ab575"} Jul 08 03:00:24.418596 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.418437 2576 scope.go:117] "RemoveContainer" containerID="cf00f67e1fbd1668277c709c5a81db76703eae38fb812ccb5923a7418cd23d66" Jul 08 03:00:24.427476 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.427447 2576 scope.go:117] "RemoveContainer" containerID="cf00f67e1fbd1668277c709c5a81db76703eae38fb812ccb5923a7418cd23d66" Jul 08 03:00:24.427745 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:00:24.427725 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf00f67e1fbd1668277c709c5a81db76703eae38fb812ccb5923a7418cd23d66\": container with ID starting with cf00f67e1fbd1668277c709c5a81db76703eae38fb812ccb5923a7418cd23d66 not found: ID does not exist" containerID="cf00f67e1fbd1668277c709c5a81db76703eae38fb812ccb5923a7418cd23d66" Jul 08 03:00:24.427790 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.427754 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf00f67e1fbd1668277c709c5a81db76703eae38fb812ccb5923a7418cd23d66"} err="failed to get container status \"cf00f67e1fbd1668277c709c5a81db76703eae38fb812ccb5923a7418cd23d66\": rpc error: code = NotFound desc = could not find container \"cf00f67e1fbd1668277c709c5a81db76703eae38fb812ccb5923a7418cd23d66\": container with ID starting with cf00f67e1fbd1668277c709c5a81db76703eae38fb812ccb5923a7418cd23d66 not found: ID does not exist" Jul 08 03:00:24.439060 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.439020 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d649cd899-xxsrz"] Jul 08 03:00:24.442253 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:24.442232 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d649cd899-xxsrz"] Jul 08 03:00:25.524983 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:25.524937 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93a506de-ab52-48e5-851e-647d7624754c" path="/var/lib/kubelet/pods/93a506de-ab52-48e5-851e-647d7624754c/volumes" Jul 08 03:00:27.434060 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:27.434018 2576 generic.go:356] "Generic (PLEG): container finished" podID="f73570e2-30eb-4cb3-9d67-0b6689c9f312" containerID="d4a6139818a175c772071d9c7f217949f23d8b30e3a12582a6281ce5e0091f36" exitCode=0 Jul 08 03:00:27.434428 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:27.434094 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" event={"ID":"f73570e2-30eb-4cb3-9d67-0b6689c9f312","Type":"ContainerDied","Data":"d4a6139818a175c772071d9c7f217949f23d8b30e3a12582a6281ce5e0091f36"} Jul 08 03:00:27.434428 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:27.434402 2576 scope.go:117] "RemoveContainer" containerID="d4a6139818a175c772071d9c7f217949f23d8b30e3a12582a6281ce5e0091f36" Jul 08 03:00:28.439998 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:28.439963 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jz8tg" event={"ID":"f73570e2-30eb-4cb3-9d67-0b6689c9f312","Type":"ContainerStarted","Data":"c69edec311c8bd5835ebcf5a6bb8bc6c0d9c6ac8fca8272dc1df2652aabf43c7"} Jul 08 03:00:32.459417 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:32.459381 2576 generic.go:356] "Generic (PLEG): container finished" podID="632607e8-3768-4d7c-8c3c-7bdcc00ad8d6" containerID="24a04ab81c83cd7df7c794b9c10482da206126c611b2a2ccbcc78feb38ccb67f" exitCode=0 Jul 08 03:00:32.459794 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:32.459450 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" event={"ID":"632607e8-3768-4d7c-8c3c-7bdcc00ad8d6","Type":"ContainerDied","Data":"24a04ab81c83cd7df7c794b9c10482da206126c611b2a2ccbcc78feb38ccb67f"} Jul 08 03:00:32.459794 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:32.459731 2576 scope.go:117] "RemoveContainer" containerID="24a04ab81c83cd7df7c794b9c10482da206126c611b2a2ccbcc78feb38ccb67f" Jul 08 03:00:33.465038 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:00:33.465001 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5dkrk" event={"ID":"632607e8-3768-4d7c-8c3c-7bdcc00ad8d6","Type":"ContainerStarted","Data":"334f6a035c571a1b5c78fd4e63bd4d0c9f74b4c47efaf86df7078c0ecd11cd83"} Jul 08 03:03:01.421366 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:03:01.421330 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:03:01.426583 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:03:01.426559 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:03:01.434094 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:03:01.434037 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:03:01.438454 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:03:01.438433 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 03:03:01.440077 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:03:01.440036 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:05:38.828289 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.828211 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4"] Jul 08 03:05:38.828736 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.828509 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" containerName="registry" Jul 08 03:05:38.828736 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.828520 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" containerName="registry" Jul 08 03:05:38.828736 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.828534 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a506de-ab52-48e5-851e-647d7624754c" containerName="console" Jul 08 03:05:38.828736 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.828540 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a506de-ab52-48e5-851e-647d7624754c" containerName="console" Jul 08 03:05:38.828736 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.828548 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba6f8fe0-46b4-40bf-9455-dff1f85c8e76" containerName="console" Jul 08 03:05:38.828736 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.828554 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba6f8fe0-46b4-40bf-9455-dff1f85c8e76" containerName="console" Jul 08 03:05:38.828736 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.828631 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba6f8fe0-46b4-40bf-9455-dff1f85c8e76" containerName="console" Jul 08 03:05:38.828736 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.828641 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f7b270-4a1d-4fbb-9913-56cc0e4fd1b5" containerName="registry" Jul 08 03:05:38.828736 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.828649 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="93a506de-ab52-48e5-851e-647d7624754c" containerName="console" Jul 08 03:05:38.830476 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.830458 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:38.834685 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.834664 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 03:05:38.834893 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.834878 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 03:05:38.834992 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.834975 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 03:05:38.835334 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.835321 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-47wqr\"" Jul 08 03:05:38.835534 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.835520 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 03:05:38.846242 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.846212 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4"] Jul 08 03:05:38.968569 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.968531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28b83991-6db1-4ebf-8052-e081b7e91def-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:38.968729 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.968583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28b83991-6db1-4ebf-8052-e081b7e91def-webhook-cert\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:38.968729 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.968660 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28b83991-6db1-4ebf-8052-e081b7e91def-manifests\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:38.968729 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.968684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28b83991-6db1-4ebf-8052-e081b7e91def-tmp\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:38.968729 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:38.968706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4jvz\" (UniqueName: \"kubernetes.io/projected/28b83991-6db1-4ebf-8052-e081b7e91def-kube-api-access-w4jvz\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:39.069913 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:39.069871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28b83991-6db1-4ebf-8052-e081b7e91def-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:39.070142 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:39.069929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28b83991-6db1-4ebf-8052-e081b7e91def-webhook-cert\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:39.070142 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:39.069970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28b83991-6db1-4ebf-8052-e081b7e91def-manifests\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:39.070142 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:39.069992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28b83991-6db1-4ebf-8052-e081b7e91def-tmp\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:39.070142 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:39.070023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4jvz\" (UniqueName: \"kubernetes.io/projected/28b83991-6db1-4ebf-8052-e081b7e91def-kube-api-access-w4jvz\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:39.070516 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:39.070489 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28b83991-6db1-4ebf-8052-e081b7e91def-manifests\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:39.072445 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:39.072423 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28b83991-6db1-4ebf-8052-e081b7e91def-tmp\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:39.072765 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:39.072743 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28b83991-6db1-4ebf-8052-e081b7e91def-webhook-cert\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:39.072838 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:39.072749 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28b83991-6db1-4ebf-8052-e081b7e91def-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:39.081485 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:39.081432 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4jvz\" (UniqueName: \"kubernetes.io/projected/28b83991-6db1-4ebf-8052-e081b7e91def-kube-api-access-w4jvz\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-72rc4\" (UID: \"28b83991-6db1-4ebf-8052-e081b7e91def\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:39.142279 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:39.142244 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:39.277633 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:39.277602 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4"] Jul 08 03:05:39.281229 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:05:39.281202 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b83991_6db1_4ebf_8052_e081b7e91def.slice/crio-0431eb9fd2d86220e5153710a82473ae2a8e7d62adcf2390b1f3ad307d76a119 WatchSource:0}: Error finding container 0431eb9fd2d86220e5153710a82473ae2a8e7d62adcf2390b1f3ad307d76a119: Status 404 returned error can't find the container with id 0431eb9fd2d86220e5153710a82473ae2a8e7d62adcf2390b1f3ad307d76a119 Jul 08 03:05:39.285596 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:39.285575 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 03:05:39.653349 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:39.653312 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" event={"ID":"28b83991-6db1-4ebf-8052-e081b7e91def","Type":"ContainerStarted","Data":"0431eb9fd2d86220e5153710a82473ae2a8e7d62adcf2390b1f3ad307d76a119"} Jul 08 03:05:42.671756 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:42.671717 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" event={"ID":"28b83991-6db1-4ebf-8052-e081b7e91def","Type":"ContainerStarted","Data":"3c03caeff1c553d59acc1e6dcef68eaf39a189c379ea0c813beb270902c4c7f0"} Jul 08 03:05:43.677345 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:43.677309 2576 generic.go:356] "Generic (PLEG): container finished" podID="28b83991-6db1-4ebf-8052-e081b7e91def" containerID="3c03caeff1c553d59acc1e6dcef68eaf39a189c379ea0c813beb270902c4c7f0" exitCode=0 Jul 08 03:05:43.677744 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:43.677400 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" event={"ID":"28b83991-6db1-4ebf-8052-e081b7e91def","Type":"ContainerDied","Data":"3c03caeff1c553d59acc1e6dcef68eaf39a189c379ea0c813beb270902c4c7f0"} Jul 08 03:05:44.684832 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:44.684794 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" event={"ID":"28b83991-6db1-4ebf-8052-e081b7e91def","Type":"ContainerStarted","Data":"e93087379a3dcc6d2d3ba6ddaa54fa7efd074d262388309dca216f53bf2fd68e"} Jul 08 03:05:44.685251 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:44.684933 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:44.708451 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:44.708395 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" podStartSLOduration=2.279624569 podStartE2EDuration="6.708378323s" totalImagesPullingTime="3.396090311s" totalInitContainerRuntime="1.032663443s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:05:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:05:44.70603044 +0000 UTC m=+463.696478516" watchObservedRunningTime="2026-07-08 03:05:44.708378323 +0000 UTC m=+463.698826412" Jul 08 03:05:55.691230 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:55.691186 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-72rc4" Jul 08 03:05:59.455535 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:59.455493 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-45dln"] Jul 08 03:05:59.461451 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:59.461421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-45dln" Jul 08 03:05:59.469505 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:59.469471 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 08 03:05:59.472610 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:59.472589 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5fqs8\"" Jul 08 03:05:59.479784 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:59.479760 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-45dln"] Jul 08 03:05:59.641293 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:59.641257 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6rfx\" (UniqueName: \"kubernetes.io/projected/4604d641-e2b0-4c88-9428-a8cec39030e4-kube-api-access-n6rfx\") pod \"odh-model-controller-7998c67dc4-45dln\" (UID: \"4604d641-e2b0-4c88-9428-a8cec39030e4\") " pod="opendatahub/odh-model-controller-7998c67dc4-45dln" Jul 08 03:05:59.641293 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:59.641301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4604d641-e2b0-4c88-9428-a8cec39030e4-cert\") pod \"odh-model-controller-7998c67dc4-45dln\" (UID: \"4604d641-e2b0-4c88-9428-a8cec39030e4\") " pod="opendatahub/odh-model-controller-7998c67dc4-45dln" Jul 08 03:05:59.741957 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:59.741859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6rfx\" (UniqueName: \"kubernetes.io/projected/4604d641-e2b0-4c88-9428-a8cec39030e4-kube-api-access-n6rfx\") pod \"odh-model-controller-7998c67dc4-45dln\" (UID: \"4604d641-e2b0-4c88-9428-a8cec39030e4\") " pod="opendatahub/odh-model-controller-7998c67dc4-45dln" Jul 08 03:05:59.741957 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:59.741914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4604d641-e2b0-4c88-9428-a8cec39030e4-cert\") pod \"odh-model-controller-7998c67dc4-45dln\" (UID: \"4604d641-e2b0-4c88-9428-a8cec39030e4\") " pod="opendatahub/odh-model-controller-7998c67dc4-45dln" Jul 08 03:05:59.744951 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:59.744917 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4604d641-e2b0-4c88-9428-a8cec39030e4-cert\") pod \"odh-model-controller-7998c67dc4-45dln\" (UID: \"4604d641-e2b0-4c88-9428-a8cec39030e4\") " pod="opendatahub/odh-model-controller-7998c67dc4-45dln" Jul 08 03:05:59.764301 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:59.764272 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6rfx\" (UniqueName: \"kubernetes.io/projected/4604d641-e2b0-4c88-9428-a8cec39030e4-kube-api-access-n6rfx\") pod \"odh-model-controller-7998c67dc4-45dln\" (UID: \"4604d641-e2b0-4c88-9428-a8cec39030e4\") " pod="opendatahub/odh-model-controller-7998c67dc4-45dln" Jul 08 03:05:59.774781 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:59.774755 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-45dln" Jul 08 03:05:59.948779 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:05:59.948750 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-45dln"] Jul 08 03:05:59.952778 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:05:59.952748 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4604d641_e2b0_4c88_9428_a8cec39030e4.slice/crio-ee700f1532aa896a4b9d7ee148150db47085ee5cad6904ee64034357ca93bbc8 WatchSource:0}: Error finding container ee700f1532aa896a4b9d7ee148150db47085ee5cad6904ee64034357ca93bbc8: Status 404 returned error can't find the container with id ee700f1532aa896a4b9d7ee148150db47085ee5cad6904ee64034357ca93bbc8 Jul 08 03:06:00.320771 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.320735 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj"] Jul 08 03:06:00.323935 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.323912 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj" Jul 08 03:06:00.348591 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.348564 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 03:06:00.348831 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.348813 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 03:06:00.369879 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.369835 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj"] Jul 08 03:06:00.449658 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.449626 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2908c98-8422-474a-97d4-95f5e41cdbe1-tls-certs\") pod \"kube-auth-proxy-58b85c4fb4-kwjfj\" (UID: \"f2908c98-8422-474a-97d4-95f5e41cdbe1\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj" Jul 08 03:06:00.449812 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.449665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2lkc\" (UniqueName: \"kubernetes.io/projected/f2908c98-8422-474a-97d4-95f5e41cdbe1-kube-api-access-k2lkc\") pod \"kube-auth-proxy-58b85c4fb4-kwjfj\" (UID: \"f2908c98-8422-474a-97d4-95f5e41cdbe1\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj" Jul 08 03:06:00.449812 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.449690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2908c98-8422-474a-97d4-95f5e41cdbe1-tmp\") pod \"kube-auth-proxy-58b85c4fb4-kwjfj\" (UID: \"f2908c98-8422-474a-97d4-95f5e41cdbe1\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj" Jul 08 03:06:00.550819 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.550780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2908c98-8422-474a-97d4-95f5e41cdbe1-tls-certs\") pod \"kube-auth-proxy-58b85c4fb4-kwjfj\" (UID: \"f2908c98-8422-474a-97d4-95f5e41cdbe1\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj" Jul 08 03:06:00.551311 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.550829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2lkc\" (UniqueName: \"kubernetes.io/projected/f2908c98-8422-474a-97d4-95f5e41cdbe1-kube-api-access-k2lkc\") pod \"kube-auth-proxy-58b85c4fb4-kwjfj\" (UID: \"f2908c98-8422-474a-97d4-95f5e41cdbe1\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj" Jul 08 03:06:00.551311 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.550950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2908c98-8422-474a-97d4-95f5e41cdbe1-tmp\") pod \"kube-auth-proxy-58b85c4fb4-kwjfj\" (UID: \"f2908c98-8422-474a-97d4-95f5e41cdbe1\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj" Jul 08 03:06:00.553320 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.553299 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2908c98-8422-474a-97d4-95f5e41cdbe1-tmp\") pod \"kube-auth-proxy-58b85c4fb4-kwjfj\" (UID: \"f2908c98-8422-474a-97d4-95f5e41cdbe1\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj" Jul 08 03:06:00.553417 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.553400 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2908c98-8422-474a-97d4-95f5e41cdbe1-tls-certs\") pod \"kube-auth-proxy-58b85c4fb4-kwjfj\" (UID: \"f2908c98-8422-474a-97d4-95f5e41cdbe1\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj" Jul 08 03:06:00.565010 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.564984 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2lkc\" (UniqueName: \"kubernetes.io/projected/f2908c98-8422-474a-97d4-95f5e41cdbe1-kube-api-access-k2lkc\") pod \"kube-auth-proxy-58b85c4fb4-kwjfj\" (UID: \"f2908c98-8422-474a-97d4-95f5e41cdbe1\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj" Jul 08 03:06:00.633644 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.633563 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj" Jul 08 03:06:00.763997 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.763961 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vgv8m"] Jul 08 03:06:00.769480 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.766648 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-45dln" event={"ID":"4604d641-e2b0-4c88-9428-a8cec39030e4","Type":"ContainerStarted","Data":"ee700f1532aa896a4b9d7ee148150db47085ee5cad6904ee64034357ca93bbc8"} Jul 08 03:06:00.769480 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.766774 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vgv8m" Jul 08 03:06:00.774368 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.774240 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 08 03:06:00.784988 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.784919 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vgv8m"] Jul 08 03:06:00.847418 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.847386 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj"] Jul 08 03:06:00.850724 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:06:00.850697 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2908c98_8422_474a_97d4_95f5e41cdbe1.slice/crio-de749f44bc5002ba268b570f420b391f2d03fdd178e8bcde7edbba0a8f5227ee WatchSource:0}: Error finding container de749f44bc5002ba268b570f420b391f2d03fdd178e8bcde7edbba0a8f5227ee: Status 404 returned error can't find the container with id de749f44bc5002ba268b570f420b391f2d03fdd178e8bcde7edbba0a8f5227ee Jul 08 03:06:00.955456 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.955326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e60c26-8066-4069-9db0-7ddbecf277f1-tls-certs\") pod \"model-serving-api-54d89bd5fc-vgv8m\" (UID: \"41e60c26-8066-4069-9db0-7ddbecf277f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-vgv8m" Jul 08 03:06:00.955456 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:00.955384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw9kp\" (UniqueName: \"kubernetes.io/projected/41e60c26-8066-4069-9db0-7ddbecf277f1-kube-api-access-qw9kp\") pod \"model-serving-api-54d89bd5fc-vgv8m\" (UID: \"41e60c26-8066-4069-9db0-7ddbecf277f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-vgv8m" Jul 08 03:06:01.056559 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.056521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e60c26-8066-4069-9db0-7ddbecf277f1-tls-certs\") pod \"model-serving-api-54d89bd5fc-vgv8m\" (UID: \"41e60c26-8066-4069-9db0-7ddbecf277f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-vgv8m" Jul 08 03:06:01.056757 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.056584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw9kp\" (UniqueName: \"kubernetes.io/projected/41e60c26-8066-4069-9db0-7ddbecf277f1-kube-api-access-qw9kp\") pod \"model-serving-api-54d89bd5fc-vgv8m\" (UID: \"41e60c26-8066-4069-9db0-7ddbecf277f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-vgv8m" Jul 08 03:06:01.056757 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:06:01.056684 2576 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 08 03:06:01.056757 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:06:01.056752 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41e60c26-8066-4069-9db0-7ddbecf277f1-tls-certs podName:41e60c26-8066-4069-9db0-7ddbecf277f1 nodeName:}" failed. No retries permitted until 2026-07-08 03:06:01.556733058 +0000 UTC m=+480.547181073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/41e60c26-8066-4069-9db0-7ddbecf277f1-tls-certs") pod "model-serving-api-54d89bd5fc-vgv8m" (UID: "41e60c26-8066-4069-9db0-7ddbecf277f1") : secret "model-serving-api-tls" not found Jul 08 03:06:01.069657 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.069625 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw9kp\" (UniqueName: \"kubernetes.io/projected/41e60c26-8066-4069-9db0-7ddbecf277f1-kube-api-access-qw9kp\") pod \"model-serving-api-54d89bd5fc-vgv8m\" (UID: \"41e60c26-8066-4069-9db0-7ddbecf277f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-vgv8m" Jul 08 03:06:01.562637 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.562601 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e60c26-8066-4069-9db0-7ddbecf277f1-tls-certs\") pod \"model-serving-api-54d89bd5fc-vgv8m\" (UID: \"41e60c26-8066-4069-9db0-7ddbecf277f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-vgv8m" Jul 08 03:06:01.565029 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.565005 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e60c26-8066-4069-9db0-7ddbecf277f1-tls-certs\") pod \"model-serving-api-54d89bd5fc-vgv8m\" (UID: \"41e60c26-8066-4069-9db0-7ddbecf277f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-vgv8m" Jul 08 03:06:01.684288 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.684246 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vgv8m" Jul 08 03:06:01.777979 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.777909 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj" event={"ID":"f2908c98-8422-474a-97d4-95f5e41cdbe1","Type":"ContainerStarted","Data":"de749f44bc5002ba268b570f420b391f2d03fdd178e8bcde7edbba0a8f5227ee"} Jul 08 03:06:01.864216 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.864082 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vgv8m"] Jul 08 03:06:01.882289 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:06:01.882254 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e60c26_8066_4069_9db0_7ddbecf277f1.slice/crio-05623da52d6ef28658f0ddfd9aac44f9f952f307c69d15745a4110e1243c7ea0 WatchSource:0}: Error finding container 05623da52d6ef28658f0ddfd9aac44f9f952f307c69d15745a4110e1243c7ea0: Status 404 returned error can't find the container with id 05623da52d6ef28658f0ddfd9aac44f9f952f307c69d15745a4110e1243c7ea0 Jul 08 03:06:01.990156 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.990119 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5678t"] Jul 08 03:06:01.994270 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.994241 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:01.998793 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.998336 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 08 03:06:01.998793 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.998587 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 08 03:06:01.998793 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.998615 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tst6f\"" Jul 08 03:06:01.999022 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.998961 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 08 03:06:01.999228 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:01.999201 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 03:06:02.017262 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.017230 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5678t"] Jul 08 03:06:02.169823 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.169781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-token\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.170085 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.169852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.170085 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.169939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.170085 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.169972 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-cacerts\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.170085 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.170012 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.170085 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.170067 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llwx7\" (UniqueName: \"kubernetes.io/projected/948fcad6-f2fa-4015-a100-95cd125a46fa-kube-api-access-llwx7\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.170460 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.170100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/948fcad6-f2fa-4015-a100-95cd125a46fa-local-certs\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.271286 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.271253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/948fcad6-f2fa-4015-a100-95cd125a46fa-local-certs\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.271286 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.271295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-token\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.271487 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.271318 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.271487 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.271366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.271487 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.271387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-cacerts\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.271487 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.271408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.271487 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.271428 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llwx7\" (UniqueName: \"kubernetes.io/projected/948fcad6-f2fa-4015-a100-95cd125a46fa-kube-api-access-llwx7\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.272841 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.272809 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.274621 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.274570 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/948fcad6-f2fa-4015-a100-95cd125a46fa-local-certs\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.274759 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.274733 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.274828 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.274756 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-cacerts\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.276300 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.276281 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.279690 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.279668 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-token\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.280426 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.280401 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llwx7\" (UniqueName: \"kubernetes.io/projected/948fcad6-f2fa-4015-a100-95cd125a46fa-kube-api-access-llwx7\") pod \"istiod-openshift-gateway-7574459656-5678t\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.309104 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.309070 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:02.787094 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:02.787037 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vgv8m" event={"ID":"41e60c26-8066-4069-9db0-7ddbecf277f1","Type":"ContainerStarted","Data":"05623da52d6ef28658f0ddfd9aac44f9f952f307c69d15745a4110e1243c7ea0"} Jul 08 03:06:04.806533 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:04.806498 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 03:06:04.924004 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:04.923966 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5678t"] Jul 08 03:06:04.926927 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:06:04.926898 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948fcad6_f2fa_4015_a100_95cd125a46fa.slice/crio-7093e963dc31928f8f6195b5c24b2ac7d349445757587b277a0457e0432c8798 WatchSource:0}: Error finding container 7093e963dc31928f8f6195b5c24b2ac7d349445757587b277a0457e0432c8798: Status 404 returned error can't find the container with id 7093e963dc31928f8f6195b5c24b2ac7d349445757587b277a0457e0432c8798 Jul 08 03:06:05.807152 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:05.807114 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-45dln" event={"ID":"4604d641-e2b0-4c88-9428-a8cec39030e4","Type":"ContainerStarted","Data":"617dd8c34d00d8bb6c3b31b78551230ef7a735a3f03914307a03e5a24b5eaeae"} Jul 08 03:06:05.807643 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:05.807201 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-45dln" Jul 08 03:06:05.810073 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:05.810027 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj" event={"ID":"f2908c98-8422-474a-97d4-95f5e41cdbe1","Type":"ContainerStarted","Data":"8b1de9b4231c504c1add7981244128934be0e1ed4ad4c4b99f51904402322164"} Jul 08 03:06:05.812282 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:05.812254 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vgv8m" event={"ID":"41e60c26-8066-4069-9db0-7ddbecf277f1","Type":"ContainerStarted","Data":"1906fb31c5f02b61423613fea254ca8a17eef292b0852c6704f310447d387c03"} Jul 08 03:06:05.812474 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:05.812453 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vgv8m" Jul 08 03:06:05.813884 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:05.813843 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" event={"ID":"948fcad6-f2fa-4015-a100-95cd125a46fa","Type":"ContainerStarted","Data":"7093e963dc31928f8f6195b5c24b2ac7d349445757587b277a0457e0432c8798"} Jul 08 03:06:05.825092 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:05.825004 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-45dln" podStartSLOduration=1.9752459930000001 podStartE2EDuration="6.824987743s" totalImagesPullingTime="4.84974175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:06:05.823833482 +0000 UTC m=+484.814281520" watchObservedRunningTime="2026-07-08 03:06:05.824987743 +0000 UTC m=+484.815435781" Jul 08 03:06:05.842420 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:05.842373 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-kwjfj" podStartSLOduration=1.8919365639999999 podStartE2EDuration="5.842354454s" totalImagesPullingTime="3.95041789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:06:05.840714558 +0000 UTC m=+484.831162597" watchObservedRunningTime="2026-07-08 03:06:05.842354454 +0000 UTC m=+484.832802492" Jul 08 03:06:07.466150 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:07.466115 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:06:07.466420 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:07.466183 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:06:07.828183 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:07.828094 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" event={"ID":"948fcad6-f2fa-4015-a100-95cd125a46fa","Type":"ContainerStarted","Data":"3e5cf6b5ffbe879a1609026558771dd7cc92334512bd9633ea0d1ecc50ab944a"} Jul 08 03:06:07.828337 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:07.828235 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:07.851171 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:07.851123 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vgv8m" podStartSLOduration=4.933815058 podStartE2EDuration="7.851107765s" totalImagesPullingTime="2.917292707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:06:05.858403748 +0000 UTC m=+484.848851786" watchObservedRunningTime="2026-07-08 03:06:07.851107765 +0000 UTC m=+486.841555842" Jul 08 03:06:07.852293 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:07.852246 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" podStartSLOduration=4.316299169 podStartE2EDuration="6.852237323s" totalImagesPullingTime="2.535938154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:06:07.84940155 +0000 UTC m=+486.839849587" watchObservedRunningTime="2026-07-08 03:06:07.852237323 +0000 UTC m=+486.842685359" Jul 08 03:06:08.834950 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:08.834917 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-5678t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 03:06:08.835323 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:08.834975 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" podUID="948fcad6-f2fa-4015-a100-95cd125a46fa" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 03:06:11.434743 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:11.434712 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj"] Jul 08 03:06:11.438490 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:11.438469 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj" Jul 08 03:06:11.441553 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:11.441522 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wwbkm\"" Jul 08 03:06:11.441861 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:11.441845 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 08 03:06:11.450936 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:11.450913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90eb5f58-6d3f-4353-8d61-d6cf7891efed-cert\") pod \"kserve-controller-manager-d6f4b65b8-mdlnj\" (UID: \"90eb5f58-6d3f-4353-8d61-d6cf7891efed\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj" Jul 08 03:06:11.451062 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:11.450950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wphlw\" (UniqueName: \"kubernetes.io/projected/90eb5f58-6d3f-4353-8d61-d6cf7891efed-kube-api-access-wphlw\") pod \"kserve-controller-manager-d6f4b65b8-mdlnj\" (UID: \"90eb5f58-6d3f-4353-8d61-d6cf7891efed\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj" Jul 08 03:06:11.455161 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:11.455140 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj"] Jul 08 03:06:11.551424 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:11.551382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90eb5f58-6d3f-4353-8d61-d6cf7891efed-cert\") pod \"kserve-controller-manager-d6f4b65b8-mdlnj\" (UID: \"90eb5f58-6d3f-4353-8d61-d6cf7891efed\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj" Jul 08 03:06:11.551424 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:11.551422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wphlw\" (UniqueName: \"kubernetes.io/projected/90eb5f58-6d3f-4353-8d61-d6cf7891efed-kube-api-access-wphlw\") pod \"kserve-controller-manager-d6f4b65b8-mdlnj\" (UID: \"90eb5f58-6d3f-4353-8d61-d6cf7891efed\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj" Jul 08 03:06:11.551684 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:06:11.551503 2576 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 08 03:06:11.551684 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:06:11.551572 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90eb5f58-6d3f-4353-8d61-d6cf7891efed-cert podName:90eb5f58-6d3f-4353-8d61-d6cf7891efed nodeName:}" failed. No retries permitted until 2026-07-08 03:06:12.051557132 +0000 UTC m=+491.042005150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/90eb5f58-6d3f-4353-8d61-d6cf7891efed-cert") pod "kserve-controller-manager-d6f4b65b8-mdlnj" (UID: "90eb5f58-6d3f-4353-8d61-d6cf7891efed") : secret "kserve-webhook-server-cert" not found Jul 08 03:06:11.561612 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:11.561574 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wphlw\" (UniqueName: \"kubernetes.io/projected/90eb5f58-6d3f-4353-8d61-d6cf7891efed-kube-api-access-wphlw\") pod \"kserve-controller-manager-d6f4b65b8-mdlnj\" (UID: \"90eb5f58-6d3f-4353-8d61-d6cf7891efed\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj" Jul 08 03:06:11.834691 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:11.834601 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:12.014929 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.014896 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5678t"] Jul 08 03:06:12.015110 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.015070 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" podUID="948fcad6-f2fa-4015-a100-95cd125a46fa" containerName="discovery" containerID="cri-o://3e5cf6b5ffbe879a1609026558771dd7cc92334512bd9633ea0d1ecc50ab944a" gracePeriod=30 Jul 08 03:06:12.056956 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.056928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90eb5f58-6d3f-4353-8d61-d6cf7891efed-cert\") pod \"kserve-controller-manager-d6f4b65b8-mdlnj\" (UID: \"90eb5f58-6d3f-4353-8d61-d6cf7891efed\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj" Jul 08 03:06:12.059967 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.059947 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90eb5f58-6d3f-4353-8d61-d6cf7891efed-cert\") pod \"kserve-controller-manager-d6f4b65b8-mdlnj\" (UID: \"90eb5f58-6d3f-4353-8d61-d6cf7891efed\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj" Jul 08 03:06:12.253506 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.253484 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:12.350183 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.350147 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj" Jul 08 03:06:12.359179 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.359160 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-csr-dns-cert\") pod \"948fcad6-f2fa-4015-a100-95cd125a46fa\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " Jul 08 03:06:12.359264 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.359200 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-kubeconfig\") pod \"948fcad6-f2fa-4015-a100-95cd125a46fa\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " Jul 08 03:06:12.359264 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.359222 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-cacerts\") pod \"948fcad6-f2fa-4015-a100-95cd125a46fa\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " Jul 08 03:06:12.359264 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.359243 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/948fcad6-f2fa-4015-a100-95cd125a46fa-local-certs\") pod \"948fcad6-f2fa-4015-a100-95cd125a46fa\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " Jul 08 03:06:12.359393 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.359289 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-token\") pod \"948fcad6-f2fa-4015-a100-95cd125a46fa\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " Jul 08 03:06:12.359393 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.359322 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llwx7\" (UniqueName: \"kubernetes.io/projected/948fcad6-f2fa-4015-a100-95cd125a46fa-kube-api-access-llwx7\") pod \"948fcad6-f2fa-4015-a100-95cd125a46fa\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " Jul 08 03:06:12.359393 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.359368 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-csr-ca-configmap\") pod \"948fcad6-f2fa-4015-a100-95cd125a46fa\" (UID: \"948fcad6-f2fa-4015-a100-95cd125a46fa\") " Jul 08 03:06:12.360206 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.360174 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "948fcad6-f2fa-4015-a100-95cd125a46fa" (UID: "948fcad6-f2fa-4015-a100-95cd125a46fa"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:06:12.361907 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.361864 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-cacerts" (OuterVolumeSpecName: "cacerts") pod "948fcad6-f2fa-4015-a100-95cd125a46fa" (UID: "948fcad6-f2fa-4015-a100-95cd125a46fa"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:06:12.362067 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.362027 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "948fcad6-f2fa-4015-a100-95cd125a46fa" (UID: "948fcad6-f2fa-4015-a100-95cd125a46fa"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:06:12.362187 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.362155 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "948fcad6-f2fa-4015-a100-95cd125a46fa" (UID: "948fcad6-f2fa-4015-a100-95cd125a46fa"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:06:12.362323 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.362240 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948fcad6-f2fa-4015-a100-95cd125a46fa-kube-api-access-llwx7" (OuterVolumeSpecName: "kube-api-access-llwx7") pod "948fcad6-f2fa-4015-a100-95cd125a46fa" (UID: "948fcad6-f2fa-4015-a100-95cd125a46fa"). InnerVolumeSpecName "kube-api-access-llwx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:06:12.362382 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.362361 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948fcad6-f2fa-4015-a100-95cd125a46fa-local-certs" (OuterVolumeSpecName: "local-certs") pod "948fcad6-f2fa-4015-a100-95cd125a46fa" (UID: "948fcad6-f2fa-4015-a100-95cd125a46fa"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:06:12.362479 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.362451 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "948fcad6-f2fa-4015-a100-95cd125a46fa" (UID: "948fcad6-f2fa-4015-a100-95cd125a46fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:06:12.460918 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.460886 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-csr-dns-cert\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:06:12.460918 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.460920 2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-kubeconfig\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:06:12.461429 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.460934 2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/948fcad6-f2fa-4015-a100-95cd125a46fa-cacerts\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:06:12.461429 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.460946 2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/948fcad6-f2fa-4015-a100-95cd125a46fa-local-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:06:12.461429 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.460958 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-token\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:06:12.461429 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.460971 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llwx7\" (UniqueName: \"kubernetes.io/projected/948fcad6-f2fa-4015-a100-95cd125a46fa-kube-api-access-llwx7\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:06:12.461429 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.460984 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/948fcad6-f2fa-4015-a100-95cd125a46fa-istio-csr-ca-configmap\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:06:12.518977 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.518951 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj"] Jul 08 03:06:12.520244 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:06:12.520222 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90eb5f58_6d3f_4353_8d61_d6cf7891efed.slice/crio-7885cca38e0e8d872312dce38578209b577558ffb08e30c44667d202dc5f4bb7 WatchSource:0}: Error finding container 7885cca38e0e8d872312dce38578209b577558ffb08e30c44667d202dc5f4bb7: Status 404 returned error can't find the container with id 7885cca38e0e8d872312dce38578209b577558ffb08e30c44667d202dc5f4bb7 Jul 08 03:06:12.852917 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.852883 2576 generic.go:356] "Generic (PLEG): container finished" podID="948fcad6-f2fa-4015-a100-95cd125a46fa" containerID="3e5cf6b5ffbe879a1609026558771dd7cc92334512bd9633ea0d1ecc50ab944a" exitCode=0 Jul 08 03:06:12.853104 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.852929 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" Jul 08 03:06:12.853104 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.852967 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" event={"ID":"948fcad6-f2fa-4015-a100-95cd125a46fa","Type":"ContainerDied","Data":"3e5cf6b5ffbe879a1609026558771dd7cc92334512bd9633ea0d1ecc50ab944a"} Jul 08 03:06:12.853104 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.853032 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5678t" event={"ID":"948fcad6-f2fa-4015-a100-95cd125a46fa","Type":"ContainerDied","Data":"7093e963dc31928f8f6195b5c24b2ac7d349445757587b277a0457e0432c8798"} Jul 08 03:06:12.853104 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.853096 2576 scope.go:117] "RemoveContainer" containerID="3e5cf6b5ffbe879a1609026558771dd7cc92334512bd9633ea0d1ecc50ab944a" Jul 08 03:06:12.854350 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.854295 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj" event={"ID":"90eb5f58-6d3f-4353-8d61-d6cf7891efed","Type":"ContainerStarted","Data":"7885cca38e0e8d872312dce38578209b577558ffb08e30c44667d202dc5f4bb7"} Jul 08 03:06:12.862961 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.862870 2576 scope.go:117] "RemoveContainer" containerID="3e5cf6b5ffbe879a1609026558771dd7cc92334512bd9633ea0d1ecc50ab944a" Jul 08 03:06:12.863270 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:06:12.863203 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5cf6b5ffbe879a1609026558771dd7cc92334512bd9633ea0d1ecc50ab944a\": container with ID starting with 3e5cf6b5ffbe879a1609026558771dd7cc92334512bd9633ea0d1ecc50ab944a not found: ID does not exist" containerID="3e5cf6b5ffbe879a1609026558771dd7cc92334512bd9633ea0d1ecc50ab944a" Jul 08 03:06:12.863270 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.863238 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5cf6b5ffbe879a1609026558771dd7cc92334512bd9633ea0d1ecc50ab944a"} err="failed to get container status \"3e5cf6b5ffbe879a1609026558771dd7cc92334512bd9633ea0d1ecc50ab944a\": rpc error: code = NotFound desc = could not find container \"3e5cf6b5ffbe879a1609026558771dd7cc92334512bd9633ea0d1ecc50ab944a\": container with ID starting with 3e5cf6b5ffbe879a1609026558771dd7cc92334512bd9633ea0d1ecc50ab944a not found: ID does not exist" Jul 08 03:06:12.899374 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.899350 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5678t"] Jul 08 03:06:12.916432 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.916399 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5678t"] Jul 08 03:06:12.944483 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.944454 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-565c746b98-6n4g2"] Jul 08 03:06:12.944865 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.944838 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="948fcad6-f2fa-4015-a100-95cd125a46fa" containerName="discovery" Jul 08 03:06:12.944865 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.944858 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="948fcad6-f2fa-4015-a100-95cd125a46fa" containerName="discovery" Jul 08 03:06:12.945007 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.944938 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="948fcad6-f2fa-4015-a100-95cd125a46fa" containerName="discovery" Jul 08 03:06:12.949938 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.949915 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:12.957807 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.952907 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bh9f8\"" Jul 08 03:06:12.957807 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.953248 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 03:06:12.957807 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.953257 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 03:06:12.957807 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.953310 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 03:06:12.957807 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.953611 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 03:06:12.957807 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.954129 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 03:06:12.961166 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.961144 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 08 03:06:12.965824 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.965799 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04c9ac21-d891-4dbc-86d9-0b236ef06072-console-config\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:12.965961 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.965866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04c9ac21-d891-4dbc-86d9-0b236ef06072-service-ca\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:12.965961 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.965893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04c9ac21-d891-4dbc-86d9-0b236ef06072-oauth-serving-cert\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:12.965961 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.965925 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04c9ac21-d891-4dbc-86d9-0b236ef06072-console-serving-cert\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:12.966186 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.965973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04c9ac21-d891-4dbc-86d9-0b236ef06072-console-oauth-config\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:12.966186 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.966034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtwrh\" (UniqueName: \"kubernetes.io/projected/04c9ac21-d891-4dbc-86d9-0b236ef06072-kube-api-access-xtwrh\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:12.966186 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.966131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04c9ac21-d891-4dbc-86d9-0b236ef06072-trusted-ca-bundle\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:12.969194 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:12.969172 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565c746b98-6n4g2"] Jul 08 03:06:13.066709 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.066675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtwrh\" (UniqueName: \"kubernetes.io/projected/04c9ac21-d891-4dbc-86d9-0b236ef06072-kube-api-access-xtwrh\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.066920 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.066751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04c9ac21-d891-4dbc-86d9-0b236ef06072-trusted-ca-bundle\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.066920 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.066794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04c9ac21-d891-4dbc-86d9-0b236ef06072-console-config\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.066920 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.066844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04c9ac21-d891-4dbc-86d9-0b236ef06072-service-ca\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.066920 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.066872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04c9ac21-d891-4dbc-86d9-0b236ef06072-oauth-serving-cert\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.066920 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.066902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04c9ac21-d891-4dbc-86d9-0b236ef06072-console-serving-cert\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.067173 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.066928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04c9ac21-d891-4dbc-86d9-0b236ef06072-console-oauth-config\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.067809 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.067764 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04c9ac21-d891-4dbc-86d9-0b236ef06072-console-config\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.067809 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.067787 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04c9ac21-d891-4dbc-86d9-0b236ef06072-service-ca\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.067809 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.067767 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04c9ac21-d891-4dbc-86d9-0b236ef06072-oauth-serving-cert\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.068368 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.068343 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04c9ac21-d891-4dbc-86d9-0b236ef06072-trusted-ca-bundle\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.069677 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.069657 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04c9ac21-d891-4dbc-86d9-0b236ef06072-console-oauth-config\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.069782 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.069767 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04c9ac21-d891-4dbc-86d9-0b236ef06072-console-serving-cert\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.077076 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.077026 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtwrh\" (UniqueName: \"kubernetes.io/projected/04c9ac21-d891-4dbc-86d9-0b236ef06072-kube-api-access-xtwrh\") pod \"console-565c746b98-6n4g2\" (UID: \"04c9ac21-d891-4dbc-86d9-0b236ef06072\") " pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.263269 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.263240 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:13.427725 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.427696 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565c746b98-6n4g2"] Jul 08 03:06:13.429927 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:06:13.429900 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c9ac21_d891_4dbc_86d9_0b236ef06072.slice/crio-fa511ff1d6764362ae935011b65a1e760afe0d34fd1ff20a26e1f97a2a4d1b91 WatchSource:0}: Error finding container fa511ff1d6764362ae935011b65a1e760afe0d34fd1ff20a26e1f97a2a4d1b91: Status 404 returned error can't find the container with id fa511ff1d6764362ae935011b65a1e760afe0d34fd1ff20a26e1f97a2a4d1b91 Jul 08 03:06:13.526950 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.526863 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="948fcad6-f2fa-4015-a100-95cd125a46fa" path="/var/lib/kubelet/pods/948fcad6-f2fa-4015-a100-95cd125a46fa/volumes" Jul 08 03:06:13.863984 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.863882 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565c746b98-6n4g2" event={"ID":"04c9ac21-d891-4dbc-86d9-0b236ef06072","Type":"ContainerStarted","Data":"58cfebed57aabe57a1f72936c7f41271c8d01234fcb35545ba55cd7b57b89e8e"} Jul 08 03:06:13.863984 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.863953 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565c746b98-6n4g2" event={"ID":"04c9ac21-d891-4dbc-86d9-0b236ef06072","Type":"ContainerStarted","Data":"fa511ff1d6764362ae935011b65a1e760afe0d34fd1ff20a26e1f97a2a4d1b91"} Jul 08 03:06:13.903889 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:13.903822 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-565c746b98-6n4g2" podStartSLOduration=1.9038041780000001 podStartE2EDuration="1.903804178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:06:13.901153235 +0000 UTC m=+492.891601271" watchObservedRunningTime="2026-07-08 03:06:13.903804178 +0000 UTC m=+492.894252215" Jul 08 03:06:15.877302 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:15.877260 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj" event={"ID":"90eb5f58-6d3f-4353-8d61-d6cf7891efed","Type":"ContainerStarted","Data":"eadd35c2a97f7c64fc7df989ea6766c4262449a96b0774d558831cb25f7b32b3"} Jul 08 03:06:15.877741 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:15.877457 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj" Jul 08 03:06:16.823645 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:16.823614 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-45dln" Jul 08 03:06:16.825668 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:16.825646 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vgv8m" Jul 08 03:06:16.841091 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:16.841013 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj" podStartSLOduration=3.211996314 podStartE2EDuration="5.84099901s" totalImagesPullingTime="2.629002696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:06:15.914114085 +0000 UTC m=+494.904562123" watchObservedRunningTime="2026-07-08 03:06:16.84099901 +0000 UTC m=+495.831447047" Jul 08 03:06:23.263686 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:23.263651 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:23.264126 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:23.263724 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:23.268882 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:23.268859 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:23.924955 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:23.924924 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-565c746b98-6n4g2" Jul 08 03:06:46.889203 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:46.889166 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mdlnj" Jul 08 03:06:59.887924 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:59.887833 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q55jv"] Jul 08 03:06:59.889990 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:59.889973 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q55jv" Jul 08 03:06:59.893463 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:59.893425 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-7df82\"" Jul 08 03:06:59.894322 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:59.894296 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 03:06:59.894463 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:59.894350 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 03:06:59.904848 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:59.904806 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q55jv"] Jul 08 03:06:59.972150 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:06:59.972121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxjlm\" (UniqueName: \"kubernetes.io/projected/5b22e34b-3520-420c-b9b5-27afc816fa7a-kube-api-access-hxjlm\") pod \"kuadrant-operator-catalog-q55jv\" (UID: \"5b22e34b-3520-420c-b9b5-27afc816fa7a\") " pod="kuadrant-system/kuadrant-operator-catalog-q55jv" Jul 08 03:07:00.072694 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:00.072658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxjlm\" (UniqueName: \"kubernetes.io/projected/5b22e34b-3520-420c-b9b5-27afc816fa7a-kube-api-access-hxjlm\") pod \"kuadrant-operator-catalog-q55jv\" (UID: \"5b22e34b-3520-420c-b9b5-27afc816fa7a\") " pod="kuadrant-system/kuadrant-operator-catalog-q55jv" Jul 08 03:07:00.081218 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:00.081188 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxjlm\" (UniqueName: \"kubernetes.io/projected/5b22e34b-3520-420c-b9b5-27afc816fa7a-kube-api-access-hxjlm\") pod \"kuadrant-operator-catalog-q55jv\" (UID: \"5b22e34b-3520-420c-b9b5-27afc816fa7a\") " pod="kuadrant-system/kuadrant-operator-catalog-q55jv" Jul 08 03:07:00.200388 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:00.200350 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q55jv" Jul 08 03:07:00.339221 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:00.339194 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q55jv"] Jul 08 03:07:00.341989 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:07:00.341960 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b22e34b_3520_420c_b9b5_27afc816fa7a.slice/crio-2a586ddafbb03c798167e67f42840e9d2457817c410e1411979466319aff4433 WatchSource:0}: Error finding container 2a586ddafbb03c798167e67f42840e9d2457817c410e1411979466319aff4433: Status 404 returned error can't find the container with id 2a586ddafbb03c798167e67f42840e9d2457817c410e1411979466319aff4433 Jul 08 03:07:00.412093 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:00.412056 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q55jv"] Jul 08 03:07:00.463034 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:00.462951 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sgp7l"] Jul 08 03:07:00.465860 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:00.465839 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sgp7l" Jul 08 03:07:00.474612 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:00.474591 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sgp7l"] Jul 08 03:07:00.576890 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:00.576850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hjvk\" (UniqueName: \"kubernetes.io/projected/d1d03998-4db4-4eaa-b0ae-8972c76cbaa9-kube-api-access-7hjvk\") pod \"kuadrant-operator-catalog-sgp7l\" (UID: \"d1d03998-4db4-4eaa-b0ae-8972c76cbaa9\") " pod="kuadrant-system/kuadrant-operator-catalog-sgp7l" Jul 08 03:07:00.677999 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:00.677952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hjvk\" (UniqueName: \"kubernetes.io/projected/d1d03998-4db4-4eaa-b0ae-8972c76cbaa9-kube-api-access-7hjvk\") pod \"kuadrant-operator-catalog-sgp7l\" (UID: \"d1d03998-4db4-4eaa-b0ae-8972c76cbaa9\") " pod="kuadrant-system/kuadrant-operator-catalog-sgp7l" Jul 08 03:07:00.686265 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:00.686234 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hjvk\" (UniqueName: \"kubernetes.io/projected/d1d03998-4db4-4eaa-b0ae-8972c76cbaa9-kube-api-access-7hjvk\") pod \"kuadrant-operator-catalog-sgp7l\" (UID: \"d1d03998-4db4-4eaa-b0ae-8972c76cbaa9\") " pod="kuadrant-system/kuadrant-operator-catalog-sgp7l" Jul 08 03:07:00.776127 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:00.776005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sgp7l" Jul 08 03:07:00.925645 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:00.925618 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sgp7l"] Jul 08 03:07:00.949331 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:07:00.949296 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d03998_4db4_4eaa_b0ae_8972c76cbaa9.slice/crio-a4d100adad330f7510a39553a68dd760bb981228ab7ee141ce4e535f5383c8a4 WatchSource:0}: Error finding container a4d100adad330f7510a39553a68dd760bb981228ab7ee141ce4e535f5383c8a4: Status 404 returned error can't find the container with id a4d100adad330f7510a39553a68dd760bb981228ab7ee141ce4e535f5383c8a4 Jul 08 03:07:01.083100 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:01.082985 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sgp7l" event={"ID":"d1d03998-4db4-4eaa-b0ae-8972c76cbaa9","Type":"ContainerStarted","Data":"a4d100adad330f7510a39553a68dd760bb981228ab7ee141ce4e535f5383c8a4"} Jul 08 03:07:01.084335 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:01.084298 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q55jv" event={"ID":"5b22e34b-3520-420c-b9b5-27afc816fa7a","Type":"ContainerStarted","Data":"2a586ddafbb03c798167e67f42840e9d2457817c410e1411979466319aff4433"} Jul 08 03:07:03.096097 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:03.096015 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q55jv" event={"ID":"5b22e34b-3520-420c-b9b5-27afc816fa7a","Type":"ContainerStarted","Data":"ab9d7b0d4c8830753f7f741c98218b9b4f8a25adb193935f583890fafb73b287"} Jul 08 03:07:03.096557 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:03.096086 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-q55jv" podUID="5b22e34b-3520-420c-b9b5-27afc816fa7a" containerName="registry-server" containerID="cri-o://ab9d7b0d4c8830753f7f741c98218b9b4f8a25adb193935f583890fafb73b287" gracePeriod=2 Jul 08 03:07:03.097693 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:03.097667 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sgp7l" event={"ID":"d1d03998-4db4-4eaa-b0ae-8972c76cbaa9","Type":"ContainerStarted","Data":"b5724c3d8e3d19ee74e1233f2501e07eb7edf699a2bcc0ba75fdfb5a52bd3d65"} Jul 08 03:07:03.114147 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:03.114103 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-q55jv" podStartSLOduration=1.912896567 podStartE2EDuration="4.114086627s" totalImagesPullingTime="2.20119006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:07:03.112670216 +0000 UTC m=+542.103118277" watchObservedRunningTime="2026-07-08 03:07:03.114086627 +0000 UTC m=+542.104534668" Jul 08 03:07:03.130466 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:03.130416 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-sgp7l" podStartSLOduration=1.532812507 podStartE2EDuration="3.130400706s" totalImagesPullingTime="1.597588199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:07:03.128316985 +0000 UTC m=+542.118765022" watchObservedRunningTime="2026-07-08 03:07:03.130400706 +0000 UTC m=+542.120848742" Jul 08 03:07:03.346648 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:03.346589 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q55jv" Jul 08 03:07:03.405653 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:03.405615 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxjlm\" (UniqueName: \"kubernetes.io/projected/5b22e34b-3520-420c-b9b5-27afc816fa7a-kube-api-access-hxjlm\") pod \"5b22e34b-3520-420c-b9b5-27afc816fa7a\" (UID: \"5b22e34b-3520-420c-b9b5-27afc816fa7a\") " Jul 08 03:07:03.407914 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:03.407881 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b22e34b-3520-420c-b9b5-27afc816fa7a-kube-api-access-hxjlm" (OuterVolumeSpecName: "kube-api-access-hxjlm") pod "5b22e34b-3520-420c-b9b5-27afc816fa7a" (UID: "5b22e34b-3520-420c-b9b5-27afc816fa7a"). InnerVolumeSpecName "kube-api-access-hxjlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:07:03.506944 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:03.506911 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxjlm\" (UniqueName: \"kubernetes.io/projected/5b22e34b-3520-420c-b9b5-27afc816fa7a-kube-api-access-hxjlm\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:07:04.103608 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:04.103572 2576 generic.go:356] "Generic (PLEG): container finished" podID="5b22e34b-3520-420c-b9b5-27afc816fa7a" containerID="ab9d7b0d4c8830753f7f741c98218b9b4f8a25adb193935f583890fafb73b287" exitCode=0 Jul 08 03:07:04.103608 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:04.103612 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q55jv" Jul 08 03:07:04.104193 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:04.103663 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q55jv" event={"ID":"5b22e34b-3520-420c-b9b5-27afc816fa7a","Type":"ContainerDied","Data":"ab9d7b0d4c8830753f7f741c98218b9b4f8a25adb193935f583890fafb73b287"} Jul 08 03:07:04.104193 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:04.103715 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q55jv" event={"ID":"5b22e34b-3520-420c-b9b5-27afc816fa7a","Type":"ContainerDied","Data":"2a586ddafbb03c798167e67f42840e9d2457817c410e1411979466319aff4433"} Jul 08 03:07:04.104193 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:04.103737 2576 scope.go:117] "RemoveContainer" containerID="ab9d7b0d4c8830753f7f741c98218b9b4f8a25adb193935f583890fafb73b287" Jul 08 03:07:04.112940 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:04.112923 2576 scope.go:117] "RemoveContainer" containerID="ab9d7b0d4c8830753f7f741c98218b9b4f8a25adb193935f583890fafb73b287" Jul 08 03:07:04.113292 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:04.113273 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9d7b0d4c8830753f7f741c98218b9b4f8a25adb193935f583890fafb73b287\": container with ID starting with ab9d7b0d4c8830753f7f741c98218b9b4f8a25adb193935f583890fafb73b287 not found: ID does not exist" containerID="ab9d7b0d4c8830753f7f741c98218b9b4f8a25adb193935f583890fafb73b287" Jul 08 03:07:04.113365 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:04.113300 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9d7b0d4c8830753f7f741c98218b9b4f8a25adb193935f583890fafb73b287"} err="failed to get container status \"ab9d7b0d4c8830753f7f741c98218b9b4f8a25adb193935f583890fafb73b287\": rpc error: code = NotFound desc = could not find container \"ab9d7b0d4c8830753f7f741c98218b9b4f8a25adb193935f583890fafb73b287\": container with ID starting with ab9d7b0d4c8830753f7f741c98218b9b4f8a25adb193935f583890fafb73b287 not found: ID does not exist" Jul 08 03:07:04.120414 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:04.120389 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q55jv"] Jul 08 03:07:04.125259 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:04.125238 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q55jv"] Jul 08 03:07:05.525206 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:05.525173 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b22e34b-3520-420c-b9b5-27afc816fa7a" path="/var/lib/kubelet/pods/5b22e34b-3520-420c-b9b5-27afc816fa7a/volumes" Jul 08 03:07:10.777165 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:10.777120 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-sgp7l" Jul 08 03:07:10.777165 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:10.777185 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-sgp7l" Jul 08 03:07:10.800098 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:10.800073 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-sgp7l" Jul 08 03:07:11.157900 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:11.157819 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-sgp7l" Jul 08 03:07:31.492001 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:31.491964 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l"] Jul 08 03:07:31.492479 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:31.492317 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b22e34b-3520-420c-b9b5-27afc816fa7a" containerName="registry-server" Jul 08 03:07:31.492479 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:31.492329 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b22e34b-3520-420c-b9b5-27afc816fa7a" containerName="registry-server" Jul 08 03:07:31.492479 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:31.492388 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b22e34b-3520-420c-b9b5-27afc816fa7a" containerName="registry-server" Jul 08 03:07:31.498546 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:31.498524 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:31.501518 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:31.501497 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-sfws8\"" Jul 08 03:07:31.505220 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:31.505196 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l"] Jul 08 03:07:31.658667 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:31.658638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwpw2\" (UniqueName: \"kubernetes.io/projected/f8ba7a03-d04a-4341-b530-55919244bfcf-kube-api-access-wwpw2\") pod \"limitador-operator-controller-manager-859d7b78bd-7k78l\" (UID: \"f8ba7a03-d04a-4341-b530-55919244bfcf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:31.759922 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:31.759852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwpw2\" (UniqueName: \"kubernetes.io/projected/f8ba7a03-d04a-4341-b530-55919244bfcf-kube-api-access-wwpw2\") pod \"limitador-operator-controller-manager-859d7b78bd-7k78l\" (UID: \"f8ba7a03-d04a-4341-b530-55919244bfcf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:31.771409 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:31.771372 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwpw2\" (UniqueName: \"kubernetes.io/projected/f8ba7a03-d04a-4341-b530-55919244bfcf-kube-api-access-wwpw2\") pod \"limitador-operator-controller-manager-859d7b78bd-7k78l\" (UID: \"f8ba7a03-d04a-4341-b530-55919244bfcf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:31.809593 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:31.809568 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:31.952320 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:31.950120 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l"] Jul 08 03:07:31.956847 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:07:31.956812 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ba7a03_d04a_4341_b530_55919244bfcf.slice/crio-f1cf41576d065ef6d4014789afd1d6eb60ce9624b8956f5b7edbf60b86ae0a08 WatchSource:0}: Error finding container f1cf41576d065ef6d4014789afd1d6eb60ce9624b8956f5b7edbf60b86ae0a08: Status 404 returned error can't find the container with id f1cf41576d065ef6d4014789afd1d6eb60ce9624b8956f5b7edbf60b86ae0a08 Jul 08 03:07:32.231893 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:32.231858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" event={"ID":"f8ba7a03-d04a-4341-b530-55919244bfcf","Type":"ContainerStarted","Data":"f1cf41576d065ef6d4014789afd1d6eb60ce9624b8956f5b7edbf60b86ae0a08"} Jul 08 03:07:34.244568 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:34.244532 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" event={"ID":"f8ba7a03-d04a-4341-b530-55919244bfcf","Type":"ContainerStarted","Data":"a33d7dbdff897a287d943c9c917e5916fc56f8db81f7b959fd3fa4040960386f"} Jul 08 03:07:34.244929 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:34.244617 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:34.262531 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:34.262482 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" podStartSLOduration=1.141735012 podStartE2EDuration="3.262468525s" totalImagesPullingTime="2.120733513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:07:34.261201005 +0000 UTC m=+573.251649043" watchObservedRunningTime="2026-07-08 03:07:34.262468525 +0000 UTC m=+573.252916562" Jul 08 03:07:36.672517 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:36.672475 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8"] Jul 08 03:07:36.678569 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:36.678552 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:36.682256 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:36.682233 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8bn97\"" Jul 08 03:07:36.691340 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:36.691316 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8"] Jul 08 03:07:36.805509 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:36.805471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhrzm\" (UniqueName: \"kubernetes.io/projected/737c9c90-6aa4-47d9-95ff-e37da6ed7bf2-kube-api-access-dhrzm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" (UID: \"737c9c90-6aa4-47d9-95ff-e37da6ed7bf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:36.805678 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:36.805522 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/737c9c90-6aa4-47d9-95ff-e37da6ed7bf2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" (UID: \"737c9c90-6aa4-47d9-95ff-e37da6ed7bf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:36.906558 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:36.906526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhrzm\" (UniqueName: \"kubernetes.io/projected/737c9c90-6aa4-47d9-95ff-e37da6ed7bf2-kube-api-access-dhrzm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" (UID: \"737c9c90-6aa4-47d9-95ff-e37da6ed7bf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:36.906778 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:36.906575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/737c9c90-6aa4-47d9-95ff-e37da6ed7bf2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" (UID: \"737c9c90-6aa4-47d9-95ff-e37da6ed7bf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:36.906928 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:36.906911 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/737c9c90-6aa4-47d9-95ff-e37da6ed7bf2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" (UID: \"737c9c90-6aa4-47d9-95ff-e37da6ed7bf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:36.915768 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:36.915749 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhrzm\" (UniqueName: \"kubernetes.io/projected/737c9c90-6aa4-47d9-95ff-e37da6ed7bf2-kube-api-access-dhrzm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" (UID: \"737c9c90-6aa4-47d9-95ff-e37da6ed7bf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:36.988589 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:36.988520 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:37.128144 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:37.128118 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8"] Jul 08 03:07:37.130919 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:07:37.130877 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737c9c90_6aa4_47d9_95ff_e37da6ed7bf2.slice/crio-948c0f79a6ef4c02a3e7c31890122c3f24a76beb5a4d4e4959df539882dffac0 WatchSource:0}: Error finding container 948c0f79a6ef4c02a3e7c31890122c3f24a76beb5a4d4e4959df539882dffac0: Status 404 returned error can't find the container with id 948c0f79a6ef4c02a3e7c31890122c3f24a76beb5a4d4e4959df539882dffac0 Jul 08 03:07:37.258668 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:37.258585 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" event={"ID":"737c9c90-6aa4-47d9-95ff-e37da6ed7bf2","Type":"ContainerStarted","Data":"948c0f79a6ef4c02a3e7c31890122c3f24a76beb5a4d4e4959df539882dffac0"} Jul 08 03:07:39.429063 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:39.429012 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vxtpv"] Jul 08 03:07:39.432592 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:39.432565 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vxtpv" Jul 08 03:07:39.435325 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:39.435303 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 08 03:07:39.435632 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:39.435612 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6hp9m\"" Jul 08 03:07:39.448847 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:39.448823 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vxtpv"] Jul 08 03:07:39.531850 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:39.531820 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vhf2\" (UniqueName: \"kubernetes.io/projected/09eaec25-bde3-4702-badd-b132f6d7dc2c-kube-api-access-4vhf2\") pod \"dns-operator-controller-manager-864dd78c4c-vxtpv\" (UID: \"09eaec25-bde3-4702-badd-b132f6d7dc2c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vxtpv" Jul 08 03:07:39.633219 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:39.633183 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vhf2\" (UniqueName: \"kubernetes.io/projected/09eaec25-bde3-4702-badd-b132f6d7dc2c-kube-api-access-4vhf2\") pod \"dns-operator-controller-manager-864dd78c4c-vxtpv\" (UID: \"09eaec25-bde3-4702-badd-b132f6d7dc2c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vxtpv" Jul 08 03:07:39.646814 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:39.646779 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vhf2\" (UniqueName: \"kubernetes.io/projected/09eaec25-bde3-4702-badd-b132f6d7dc2c-kube-api-access-4vhf2\") pod \"dns-operator-controller-manager-864dd78c4c-vxtpv\" (UID: \"09eaec25-bde3-4702-badd-b132f6d7dc2c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vxtpv" Jul 08 03:07:39.744673 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:39.744587 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vxtpv" Jul 08 03:07:39.922685 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:39.922661 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vxtpv"] Jul 08 03:07:39.924296 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:07:39.924271 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09eaec25_bde3_4702_badd_b132f6d7dc2c.slice/crio-d8adddf3b61fa03e41852b9cf3209ffbf9ab2ea6feceff339e0e5d12d523e4e5 WatchSource:0}: Error finding container d8adddf3b61fa03e41852b9cf3209ffbf9ab2ea6feceff339e0e5d12d523e4e5: Status 404 returned error can't find the container with id d8adddf3b61fa03e41852b9cf3209ffbf9ab2ea6feceff339e0e5d12d523e4e5 Jul 08 03:07:40.276386 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:40.276345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vxtpv" event={"ID":"09eaec25-bde3-4702-badd-b132f6d7dc2c","Type":"ContainerStarted","Data":"d8adddf3b61fa03e41852b9cf3209ffbf9ab2ea6feceff339e0e5d12d523e4e5"} Jul 08 03:07:42.861814 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:42.861779 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7fb2t"] Jul 08 03:07:42.865433 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:42.865410 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7fb2t" Jul 08 03:07:42.868304 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:42.868284 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-g7zk9\"" Jul 08 03:07:42.875499 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:42.875475 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7fb2t"] Jul 08 03:07:42.965458 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:42.965431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwt7\" (UniqueName: \"kubernetes.io/projected/bfbddb3e-5995-43a3-a7b5-1bbdd3047763-kube-api-access-qtwt7\") pod \"authorino-operator-75cc8d5cb7-7fb2t\" (UID: \"bfbddb3e-5995-43a3-a7b5-1bbdd3047763\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7fb2t" Jul 08 03:07:43.065907 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:43.065877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwt7\" (UniqueName: \"kubernetes.io/projected/bfbddb3e-5995-43a3-a7b5-1bbdd3047763-kube-api-access-qtwt7\") pod \"authorino-operator-75cc8d5cb7-7fb2t\" (UID: \"bfbddb3e-5995-43a3-a7b5-1bbdd3047763\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7fb2t" Jul 08 03:07:43.077211 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:43.077183 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwt7\" (UniqueName: \"kubernetes.io/projected/bfbddb3e-5995-43a3-a7b5-1bbdd3047763-kube-api-access-qtwt7\") pod \"authorino-operator-75cc8d5cb7-7fb2t\" (UID: \"bfbddb3e-5995-43a3-a7b5-1bbdd3047763\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7fb2t" Jul 08 03:07:43.178899 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:43.178875 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7fb2t" Jul 08 03:07:43.293989 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:43.293948 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" event={"ID":"737c9c90-6aa4-47d9-95ff-e37da6ed7bf2","Type":"ContainerStarted","Data":"2c945a98254a94635745b2d8b294d654d50691b04eb8ef9b6530b75f5d498e46"} Jul 08 03:07:43.294165 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:43.294102 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:43.295901 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:43.295878 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vxtpv" event={"ID":"09eaec25-bde3-4702-badd-b132f6d7dc2c","Type":"ContainerStarted","Data":"635df530949c598e7f76702fdce375e649295565c45302cb4f774ad1892a00d2"} Jul 08 03:07:43.296035 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:43.296015 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vxtpv" Jul 08 03:07:43.314188 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:43.314139 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" podStartSLOduration=1.327497179 podStartE2EDuration="7.31411013s" totalImagesPullingTime="5.986612951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:07:43.31298437 +0000 UTC m=+582.303432408" watchObservedRunningTime="2026-07-08 03:07:43.31411013 +0000 UTC m=+582.304558166" Jul 08 03:07:43.329780 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:43.329753 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7fb2t"] Jul 08 03:07:43.332632 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:07:43.332600 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbddb3e_5995_43a3_a7b5_1bbdd3047763.slice/crio-a83945eaa81e5e4893902040f7fd5df17c8182e2ca9d777ee8553257e3259082 WatchSource:0}: Error finding container a83945eaa81e5e4893902040f7fd5df17c8182e2ca9d777ee8553257e3259082: Status 404 returned error can't find the container with id a83945eaa81e5e4893902040f7fd5df17c8182e2ca9d777ee8553257e3259082 Jul 08 03:07:43.340406 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:43.340362 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vxtpv" podStartSLOduration=1.152207116 podStartE2EDuration="4.34034853s" totalImagesPullingTime="3.188141414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:07:43.339120037 +0000 UTC m=+582.329568075" watchObservedRunningTime="2026-07-08 03:07:43.34034853 +0000 UTC m=+582.330796568" Jul 08 03:07:44.303110 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:44.303072 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7fb2t" event={"ID":"bfbddb3e-5995-43a3-a7b5-1bbdd3047763","Type":"ContainerStarted","Data":"a83945eaa81e5e4893902040f7fd5df17c8182e2ca9d777ee8553257e3259082"} Jul 08 03:07:45.252158 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:45.252067 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:46.316372 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:46.316335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7fb2t" event={"ID":"bfbddb3e-5995-43a3-a7b5-1bbdd3047763","Type":"ContainerStarted","Data":"dec467d6d43f5f9e78196e9513a58e14aa81637cee20bb8b5f6e2c7c0567647e"} Jul 08 03:07:46.316746 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:46.316440 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7fb2t" Jul 08 03:07:46.348079 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:46.348014 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7fb2t" podStartSLOduration=1.866765843 podStartE2EDuration="4.347997474s" totalImagesPullingTime="2.481231631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:07:46.346413824 +0000 UTC m=+585.336861861" watchObservedRunningTime="2026-07-08 03:07:46.347997474 +0000 UTC m=+585.338445512" Jul 08 03:07:54.305595 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:54.305561 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vxtpv" Jul 08 03:07:54.306084 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:54.305648 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:56.016229 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.016200 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8"] Jul 08 03:07:56.016598 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.016384 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" containerName="manager" containerID="cri-o://2c945a98254a94635745b2d8b294d654d50691b04eb8ef9b6530b75f5d498e46" gracePeriod=2 Jul 08 03:07:56.029965 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.029937 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8"] Jul 08 03:07:56.038882 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.038855 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l"] Jul 08 03:07:56.039152 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.039096 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" containerName="manager" containerID="cri-o://a33d7dbdff897a287d943c9c917e5916fc56f8db81f7b959fd3fa4040960386f" gracePeriod=2 Jul 08 03:07:56.043586 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.043556 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd"] Jul 08 03:07:56.044174 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.044154 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" containerName="manager" Jul 08 03:07:56.044258 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.044176 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" containerName="manager" Jul 08 03:07:56.044316 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.044263 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" containerName="manager" Jul 08 03:07:56.048185 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.048163 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" Jul 08 03:07:56.051386 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.051352 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:56.052443 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.052414 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l"] Jul 08 03:07:56.057255 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.053980 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7k78l\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:56.057255 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.056575 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:56.069023 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.068995 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd"] Jul 08 03:07:56.082644 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.082619 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mtxfm"] Jul 08 03:07:56.083092 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.083072 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" containerName="manager" Jul 08 03:07:56.083183 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.083093 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" containerName="manager" Jul 08 03:07:56.083239 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.083191 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" containerName="manager" Jul 08 03:07:56.084767 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.084745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3833f3e4-0842-41bd-a660-13e017ed949b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-96hfd\" (UID: \"3833f3e4-0842-41bd-a660-13e017ed949b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" Jul 08 03:07:56.084874 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.084858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnbv9\" (UniqueName: \"kubernetes.io/projected/3833f3e4-0842-41bd-a660-13e017ed949b-kube-api-access-gnbv9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-96hfd\" (UID: \"3833f3e4-0842-41bd-a660-13e017ed949b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" Jul 08 03:07:56.086542 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.086423 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mtxfm" Jul 08 03:07:56.088997 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.088968 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7k78l\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:56.091332 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.091306 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:56.098553 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.098530 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mtxfm"] Jul 08 03:07:56.115264 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.115226 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29"] Jul 08 03:07:56.118747 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.118722 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" Jul 08 03:07:56.121415 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.121351 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7k78l\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:56.132542 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.132519 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29"] Jul 08 03:07:56.157296 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.157241 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:56.185791 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.185754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/383c043f-be58-435d-a239-26a03c31b9cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8df29\" (UID: \"383c043f-be58-435d-a239-26a03c31b9cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" Jul 08 03:07:56.185945 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.185871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnbv9\" (UniqueName: \"kubernetes.io/projected/3833f3e4-0842-41bd-a660-13e017ed949b-kube-api-access-gnbv9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-96hfd\" (UID: \"3833f3e4-0842-41bd-a660-13e017ed949b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" Jul 08 03:07:56.185945 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.185907 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2tmw\" (UniqueName: \"kubernetes.io/projected/5493b764-e3f9-4f06-9109-13a1543efb8b-kube-api-access-q2tmw\") pod \"limitador-operator-controller-manager-859d7b78bd-mtxfm\" (UID: \"5493b764-e3f9-4f06-9109-13a1543efb8b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mtxfm" Jul 08 03:07:56.186079 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.185949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3833f3e4-0842-41bd-a660-13e017ed949b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-96hfd\" (UID: \"3833f3e4-0842-41bd-a660-13e017ed949b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" Jul 08 03:07:56.186079 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.185993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgr9n\" (UniqueName: \"kubernetes.io/projected/383c043f-be58-435d-a239-26a03c31b9cb-kube-api-access-tgr9n\") pod \"kuadrant-operator-controller-manager-86878c769f-8df29\" (UID: \"383c043f-be58-435d-a239-26a03c31b9cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" Jul 08 03:07:56.186358 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.186342 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3833f3e4-0842-41bd-a660-13e017ed949b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-96hfd\" (UID: \"3833f3e4-0842-41bd-a660-13e017ed949b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" Jul 08 03:07:56.200728 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.200669 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnbv9\" (UniqueName: \"kubernetes.io/projected/3833f3e4-0842-41bd-a660-13e017ed949b-kube-api-access-gnbv9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-96hfd\" (UID: \"3833f3e4-0842-41bd-a660-13e017ed949b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" Jul 08 03:07:56.286644 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.286612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2tmw\" (UniqueName: \"kubernetes.io/projected/5493b764-e3f9-4f06-9109-13a1543efb8b-kube-api-access-q2tmw\") pod \"limitador-operator-controller-manager-859d7b78bd-mtxfm\" (UID: \"5493b764-e3f9-4f06-9109-13a1543efb8b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mtxfm" Jul 08 03:07:56.286798 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.286699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgr9n\" (UniqueName: \"kubernetes.io/projected/383c043f-be58-435d-a239-26a03c31b9cb-kube-api-access-tgr9n\") pod \"kuadrant-operator-controller-manager-86878c769f-8df29\" (UID: \"383c043f-be58-435d-a239-26a03c31b9cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" Jul 08 03:07:56.286798 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.286746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/383c043f-be58-435d-a239-26a03c31b9cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8df29\" (UID: \"383c043f-be58-435d-a239-26a03c31b9cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" Jul 08 03:07:56.287140 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.287120 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/383c043f-be58-435d-a239-26a03c31b9cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8df29\" (UID: \"383c043f-be58-435d-a239-26a03c31b9cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" Jul 08 03:07:56.290864 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.290842 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:56.292817 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.292801 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:56.293547 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.293525 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7k78l\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:56.296297 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.296276 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:56.298798 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.298769 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7k78l\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:56.298978 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.298962 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgr9n\" (UniqueName: \"kubernetes.io/projected/383c043f-be58-435d-a239-26a03c31b9cb-kube-api-access-tgr9n\") pod \"kuadrant-operator-controller-manager-86878c769f-8df29\" (UID: \"383c043f-be58-435d-a239-26a03c31b9cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" Jul 08 03:07:56.300098 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.300080 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2tmw\" (UniqueName: \"kubernetes.io/projected/5493b764-e3f9-4f06-9109-13a1543efb8b-kube-api-access-q2tmw\") pod \"limitador-operator-controller-manager-859d7b78bd-mtxfm\" (UID: \"5493b764-e3f9-4f06-9109-13a1543efb8b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mtxfm" Jul 08 03:07:56.300799 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.300774 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:56.365690 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.365657 2576 generic.go:356] "Generic (PLEG): container finished" podID="f8ba7a03-d04a-4341-b530-55919244bfcf" containerID="a33d7dbdff897a287d943c9c917e5916fc56f8db81f7b959fd3fa4040960386f" exitCode=0 Jul 08 03:07:56.365805 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.365704 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:56.365805 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.365759 2576 scope.go:117] "RemoveContainer" containerID="a33d7dbdff897a287d943c9c917e5916fc56f8db81f7b959fd3fa4040960386f" Jul 08 03:07:56.367174 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.367154 2576 generic.go:356] "Generic (PLEG): container finished" podID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" containerID="2c945a98254a94635745b2d8b294d654d50691b04eb8ef9b6530b75f5d498e46" exitCode=0 Jul 08 03:07:56.367254 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.367195 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:56.368272 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.368249 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7k78l\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:56.370422 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.370392 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:56.372556 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.372532 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7k78l\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:56.374512 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.374497 2576 scope.go:117] "RemoveContainer" containerID="a33d7dbdff897a287d943c9c917e5916fc56f8db81f7b959fd3fa4040960386f" Jul 08 03:07:56.374654 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.374633 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:56.374768 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.374751 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33d7dbdff897a287d943c9c917e5916fc56f8db81f7b959fd3fa4040960386f\": container with ID starting with a33d7dbdff897a287d943c9c917e5916fc56f8db81f7b959fd3fa4040960386f not found: ID does not exist" containerID="a33d7dbdff897a287d943c9c917e5916fc56f8db81f7b959fd3fa4040960386f" Jul 08 03:07:56.374812 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.374776 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33d7dbdff897a287d943c9c917e5916fc56f8db81f7b959fd3fa4040960386f"} err="failed to get container status \"a33d7dbdff897a287d943c9c917e5916fc56f8db81f7b959fd3fa4040960386f\": rpc error: code = NotFound desc = could not find container \"a33d7dbdff897a287d943c9c917e5916fc56f8db81f7b959fd3fa4040960386f\": container with ID starting with a33d7dbdff897a287d943c9c917e5916fc56f8db81f7b959fd3fa4040960386f not found: ID does not exist" Jul 08 03:07:56.374812 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.374792 2576 scope.go:117] "RemoveContainer" containerID="2c945a98254a94635745b2d8b294d654d50691b04eb8ef9b6530b75f5d498e46" Jul 08 03:07:56.382374 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.382357 2576 scope.go:117] "RemoveContainer" containerID="2c945a98254a94635745b2d8b294d654d50691b04eb8ef9b6530b75f5d498e46" Jul 08 03:07:56.382626 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.382609 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c945a98254a94635745b2d8b294d654d50691b04eb8ef9b6530b75f5d498e46\": container with ID starting with 2c945a98254a94635745b2d8b294d654d50691b04eb8ef9b6530b75f5d498e46 not found: ID does not exist" containerID="2c945a98254a94635745b2d8b294d654d50691b04eb8ef9b6530b75f5d498e46" Jul 08 03:07:56.382674 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.382631 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c945a98254a94635745b2d8b294d654d50691b04eb8ef9b6530b75f5d498e46"} err="failed to get container status \"2c945a98254a94635745b2d8b294d654d50691b04eb8ef9b6530b75f5d498e46\": rpc error: code = NotFound desc = could not find container \"2c945a98254a94635745b2d8b294d654d50691b04eb8ef9b6530b75f5d498e46\": container with ID starting with 2c945a98254a94635745b2d8b294d654d50691b04eb8ef9b6530b75f5d498e46 not found: ID does not exist" Jul 08 03:07:56.387885 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.387870 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhrzm\" (UniqueName: \"kubernetes.io/projected/737c9c90-6aa4-47d9-95ff-e37da6ed7bf2-kube-api-access-dhrzm\") pod \"737c9c90-6aa4-47d9-95ff-e37da6ed7bf2\" (UID: \"737c9c90-6aa4-47d9-95ff-e37da6ed7bf2\") " Jul 08 03:07:56.387941 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.387905 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/737c9c90-6aa4-47d9-95ff-e37da6ed7bf2-extensions-socket-volume\") pod \"737c9c90-6aa4-47d9-95ff-e37da6ed7bf2\" (UID: \"737c9c90-6aa4-47d9-95ff-e37da6ed7bf2\") " Jul 08 03:07:56.387980 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.387964 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwpw2\" (UniqueName: \"kubernetes.io/projected/f8ba7a03-d04a-4341-b530-55919244bfcf-kube-api-access-wwpw2\") pod \"f8ba7a03-d04a-4341-b530-55919244bfcf\" (UID: \"f8ba7a03-d04a-4341-b530-55919244bfcf\") " Jul 08 03:07:56.388444 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.388416 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/737c9c90-6aa4-47d9-95ff-e37da6ed7bf2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" (UID: "737c9c90-6aa4-47d9-95ff-e37da6ed7bf2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:07:56.389913 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.389893 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737c9c90-6aa4-47d9-95ff-e37da6ed7bf2-kube-api-access-dhrzm" (OuterVolumeSpecName: "kube-api-access-dhrzm") pod "737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" (UID: "737c9c90-6aa4-47d9-95ff-e37da6ed7bf2"). InnerVolumeSpecName "kube-api-access-dhrzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:07:56.389986 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.389946 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ba7a03-d04a-4341-b530-55919244bfcf-kube-api-access-wwpw2" (OuterVolumeSpecName: "kube-api-access-wwpw2") pod "f8ba7a03-d04a-4341-b530-55919244bfcf" (UID: "f8ba7a03-d04a-4341-b530-55919244bfcf"). InnerVolumeSpecName "kube-api-access-wwpw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:07:56.462439 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.462400 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" Jul 08 03:07:56.471598 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.471575 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mtxfm" Jul 08 03:07:56.480737 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.480710 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" Jul 08 03:07:56.488956 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.488929 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhrzm\" (UniqueName: \"kubernetes.io/projected/737c9c90-6aa4-47d9-95ff-e37da6ed7bf2-kube-api-access-dhrzm\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:07:56.488956 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.488956 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/737c9c90-6aa4-47d9-95ff-e37da6ed7bf2-extensions-socket-volume\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:07:56.489115 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.488967 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwpw2\" (UniqueName: \"kubernetes.io/projected/f8ba7a03-d04a-4341-b530-55919244bfcf-kube-api-access-wwpw2\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:07:56.648453 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.648429 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mtxfm"] Jul 08 03:07:56.651382 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:07:56.651346 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5493b764_e3f9_4f06_9109_13a1543efb8b.slice/crio-a26a75c3c5a300513bc4546c2738b360e041ea1d51c763cb7e355a4bfd6b2c59 WatchSource:0}: Error finding container a26a75c3c5a300513bc4546c2738b360e041ea1d51c763cb7e355a4bfd6b2c59: Status 404 returned error can't find the container with id a26a75c3c5a300513bc4546c2738b360e041ea1d51c763cb7e355a4bfd6b2c59 Jul 08 03:07:56.689991 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.689957 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:56.692428 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.692402 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7k78l\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:56.695896 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.695873 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7k78l\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:56.698287 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:56.698261 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:56.890718 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.890578 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29"] Jul 08 03:07:56.893131 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:56.893107 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd"] Jul 08 03:07:56.894004 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:07:56.893977 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383c043f_be58_435d_a239_26a03c31b9cb.slice/crio-ff4638edc770bcdd57a502307f90758636323f24cb92f97ef3d0a8a36eaa9779 WatchSource:0}: Error finding container ff4638edc770bcdd57a502307f90758636323f24cb92f97ef3d0a8a36eaa9779: Status 404 returned error can't find the container with id ff4638edc770bcdd57a502307f90758636323f24cb92f97ef3d0a8a36eaa9779 Jul 08 03:07:56.895728 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:07:56.895706 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3833f3e4_0842_41bd_a660_13e017ed949b.slice/crio-b1376e5ddaae561dd57dda6db838b371fb64d1d2988774f2f7c5dc2ff26dad06 WatchSource:0}: Error finding container b1376e5ddaae561dd57dda6db838b371fb64d1d2988774f2f7c5dc2ff26dad06: Status 404 returned error can't find the container with id b1376e5ddaae561dd57dda6db838b371fb64d1d2988774f2f7c5dc2ff26dad06 Jul 08 03:07:57.323409 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.323379 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7fb2t" Jul 08 03:07:57.347211 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:57.347171 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7k78l\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:57.349237 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:57.349211 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:57.375899 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.375855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mtxfm" event={"ID":"5493b764-e3f9-4f06-9109-13a1543efb8b","Type":"ContainerStarted","Data":"e24e8640891d73f528602b194a66e7730a1cca52b0cbdd67c8c05bee4b293b3b"} Jul 08 03:07:57.376081 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.375925 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mtxfm" event={"ID":"5493b764-e3f9-4f06-9109-13a1543efb8b","Type":"ContainerStarted","Data":"a26a75c3c5a300513bc4546c2738b360e041ea1d51c763cb7e355a4bfd6b2c59"} Jul 08 03:07:57.376081 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.375955 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mtxfm" Jul 08 03:07:57.377828 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.377801 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" event={"ID":"383c043f-be58-435d-a239-26a03c31b9cb","Type":"ContainerStarted","Data":"3271a85b5abb0915bad2f2e0d6201c175cd32937eb9ed76d05c57a5b577db8cc"} Jul 08 03:07:57.377949 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.377851 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" event={"ID":"383c043f-be58-435d-a239-26a03c31b9cb","Type":"ContainerStarted","Data":"ff4638edc770bcdd57a502307f90758636323f24cb92f97ef3d0a8a36eaa9779"} Jul 08 03:07:57.377949 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.377879 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" Jul 08 03:07:57.379199 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:57.379172 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7k78l\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:57.379719 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.379694 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" event={"ID":"3833f3e4-0842-41bd-a660-13e017ed949b","Type":"ContainerStarted","Data":"dc4827c1d44ddf4c9dfb610e05dfbf4d8caf67fa3f32c4ccf7ebcbff5559edbd"} Jul 08 03:07:57.379825 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.379744 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" event={"ID":"3833f3e4-0842-41bd-a660-13e017ed949b","Type":"ContainerStarted","Data":"b1376e5ddaae561dd57dda6db838b371fb64d1d2988774f2f7c5dc2ff26dad06"} Jul 08 03:07:57.379825 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.379790 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" Jul 08 03:07:57.381612 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:57.381587 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:57.402568 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.402520 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mtxfm" podStartSLOduration=1.402505528 podStartE2EDuration="1.402505528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:07:57.40081186 +0000 UTC m=+596.391259896" watchObservedRunningTime="2026-07-08 03:07:57.402505528 +0000 UTC m=+596.392953566" Jul 08 03:07:57.403437 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:57.403417 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7k78l\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7k78l" Jul 08 03:07:57.434023 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.433975 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" podStartSLOduration=1.433959617 podStartE2EDuration="1.433959617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:07:57.431786836 +0000 UTC m=+596.422234874" watchObservedRunningTime="2026-07-08 03:07:57.433959617 +0000 UTC m=+596.424407654" Jul 08 03:07:57.436079 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:07:57.436030 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vgzg8\" is forbidden: User \"system:node:ip-10-0-142-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-99.ec2.internal' and this object" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vgzg8" Jul 08 03:07:57.470466 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.470416 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" podStartSLOduration=1.470401074 podStartE2EDuration="1.470401074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:07:57.46850842 +0000 UTC m=+596.458956457" watchObservedRunningTime="2026-07-08 03:07:57.470401074 +0000 UTC m=+596.460849110" Jul 08 03:07:57.525914 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.525882 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="737c9c90-6aa4-47d9-95ff-e37da6ed7bf2" path="/var/lib/kubelet/pods/737c9c90-6aa4-47d9-95ff-e37da6ed7bf2/volumes" Jul 08 03:07:57.526233 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:07:57.526219 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8ba7a03-d04a-4341-b530-55919244bfcf" path="/var/lib/kubelet/pods/f8ba7a03-d04a-4341-b530-55919244bfcf/volumes" Jul 08 03:08:01.461750 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:01.461722 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:08:01.464516 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:01.464490 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:08:01.467070 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:01.467035 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:08:01.469749 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:01.469729 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:08:08.388216 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:08.388186 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" Jul 08 03:08:08.388770 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:08.388269 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mtxfm" Jul 08 03:08:08.388770 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:08.388315 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" Jul 08 03:08:08.498629 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:08.498602 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd"] Jul 08 03:08:08.498827 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:08.498781 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" podUID="3833f3e4-0842-41bd-a660-13e017ed949b" containerName="manager" containerID="cri-o://dc4827c1d44ddf4c9dfb610e05dfbf4d8caf67fa3f32c4ccf7ebcbff5559edbd" gracePeriod=10 Jul 08 03:08:08.766315 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:08.766289 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" Jul 08 03:08:08.900291 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:08.900256 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnbv9\" (UniqueName: \"kubernetes.io/projected/3833f3e4-0842-41bd-a660-13e017ed949b-kube-api-access-gnbv9\") pod \"3833f3e4-0842-41bd-a660-13e017ed949b\" (UID: \"3833f3e4-0842-41bd-a660-13e017ed949b\") " Jul 08 03:08:08.900291 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:08.900298 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3833f3e4-0842-41bd-a660-13e017ed949b-extensions-socket-volume\") pod \"3833f3e4-0842-41bd-a660-13e017ed949b\" (UID: \"3833f3e4-0842-41bd-a660-13e017ed949b\") " Jul 08 03:08:08.900739 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:08.900698 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3833f3e4-0842-41bd-a660-13e017ed949b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3833f3e4-0842-41bd-a660-13e017ed949b" (UID: "3833f3e4-0842-41bd-a660-13e017ed949b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:08:08.902479 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:08.902452 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3833f3e4-0842-41bd-a660-13e017ed949b-kube-api-access-gnbv9" (OuterVolumeSpecName: "kube-api-access-gnbv9") pod "3833f3e4-0842-41bd-a660-13e017ed949b" (UID: "3833f3e4-0842-41bd-a660-13e017ed949b"). InnerVolumeSpecName "kube-api-access-gnbv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:08:09.001280 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:09.001194 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnbv9\" (UniqueName: \"kubernetes.io/projected/3833f3e4-0842-41bd-a660-13e017ed949b-kube-api-access-gnbv9\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:08:09.001280 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:09.001222 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3833f3e4-0842-41bd-a660-13e017ed949b-extensions-socket-volume\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:08:09.443238 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:09.443199 2576 generic.go:356] "Generic (PLEG): container finished" podID="3833f3e4-0842-41bd-a660-13e017ed949b" containerID="dc4827c1d44ddf4c9dfb610e05dfbf4d8caf67fa3f32c4ccf7ebcbff5559edbd" exitCode=0 Jul 08 03:08:09.443642 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:09.443260 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" Jul 08 03:08:09.443642 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:09.443285 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" event={"ID":"3833f3e4-0842-41bd-a660-13e017ed949b","Type":"ContainerDied","Data":"dc4827c1d44ddf4c9dfb610e05dfbf4d8caf67fa3f32c4ccf7ebcbff5559edbd"} Jul 08 03:08:09.443642 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:09.443374 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd" event={"ID":"3833f3e4-0842-41bd-a660-13e017ed949b","Type":"ContainerDied","Data":"b1376e5ddaae561dd57dda6db838b371fb64d1d2988774f2f7c5dc2ff26dad06"} Jul 08 03:08:09.443642 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:09.443402 2576 scope.go:117] "RemoveContainer" containerID="dc4827c1d44ddf4c9dfb610e05dfbf4d8caf67fa3f32c4ccf7ebcbff5559edbd" Jul 08 03:08:09.452586 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:09.452563 2576 scope.go:117] "RemoveContainer" containerID="dc4827c1d44ddf4c9dfb610e05dfbf4d8caf67fa3f32c4ccf7ebcbff5559edbd" Jul 08 03:08:09.452857 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:08:09.452837 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4827c1d44ddf4c9dfb610e05dfbf4d8caf67fa3f32c4ccf7ebcbff5559edbd\": container with ID starting with dc4827c1d44ddf4c9dfb610e05dfbf4d8caf67fa3f32c4ccf7ebcbff5559edbd not found: ID does not exist" containerID="dc4827c1d44ddf4c9dfb610e05dfbf4d8caf67fa3f32c4ccf7ebcbff5559edbd" Jul 08 03:08:09.452903 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:09.452867 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4827c1d44ddf4c9dfb610e05dfbf4d8caf67fa3f32c4ccf7ebcbff5559edbd"} err="failed to get container status \"dc4827c1d44ddf4c9dfb610e05dfbf4d8caf67fa3f32c4ccf7ebcbff5559edbd\": rpc error: code = NotFound desc = could not find container \"dc4827c1d44ddf4c9dfb610e05dfbf4d8caf67fa3f32c4ccf7ebcbff5559edbd\": container with ID starting with dc4827c1d44ddf4c9dfb610e05dfbf4d8caf67fa3f32c4ccf7ebcbff5559edbd not found: ID does not exist" Jul 08 03:08:09.468317 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:09.468289 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd"] Jul 08 03:08:09.476836 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:09.476810 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-96hfd"] Jul 08 03:08:09.525356 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:09.525324 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3833f3e4-0842-41bd-a660-13e017ed949b" path="/var/lib/kubelet/pods/3833f3e4-0842-41bd-a660-13e017ed949b/volumes" Jul 08 03:08:27.755235 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.755148 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:08:27.755764 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.755740 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3833f3e4-0842-41bd-a660-13e017ed949b" containerName="manager" Jul 08 03:08:27.755861 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.755767 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3833f3e4-0842-41bd-a660-13e017ed949b" containerName="manager" Jul 08 03:08:27.755918 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.755898 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3833f3e4-0842-41bd-a660-13e017ed949b" containerName="manager" Jul 08 03:08:27.758102 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.758083 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jzq2l" Jul 08 03:08:27.760810 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.760788 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 03:08:27.760915 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.760820 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hqzwg\"" Jul 08 03:08:27.768718 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.768687 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:08:27.793542 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.793508 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:08:27.869115 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.869085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05db8d30-1a63-44fc-af00-baac2f1825bf-config-file\") pod \"limitador-limitador-54d6d988d5-jzq2l\" (UID: \"05db8d30-1a63-44fc-af00-baac2f1825bf\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jzq2l" Jul 08 03:08:27.869256 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.869136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdnnr\" (UniqueName: \"kubernetes.io/projected/05db8d30-1a63-44fc-af00-baac2f1825bf-kube-api-access-bdnnr\") pod \"limitador-limitador-54d6d988d5-jzq2l\" (UID: \"05db8d30-1a63-44fc-af00-baac2f1825bf\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jzq2l" Jul 08 03:08:27.970224 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.970190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05db8d30-1a63-44fc-af00-baac2f1825bf-config-file\") pod \"limitador-limitador-54d6d988d5-jzq2l\" (UID: \"05db8d30-1a63-44fc-af00-baac2f1825bf\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jzq2l" Jul 08 03:08:27.970445 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.970241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdnnr\" (UniqueName: \"kubernetes.io/projected/05db8d30-1a63-44fc-af00-baac2f1825bf-kube-api-access-bdnnr\") pod \"limitador-limitador-54d6d988d5-jzq2l\" (UID: \"05db8d30-1a63-44fc-af00-baac2f1825bf\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jzq2l" Jul 08 03:08:27.970794 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.970774 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05db8d30-1a63-44fc-af00-baac2f1825bf-config-file\") pod \"limitador-limitador-54d6d988d5-jzq2l\" (UID: \"05db8d30-1a63-44fc-af00-baac2f1825bf\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jzq2l" Jul 08 03:08:27.979396 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:27.979366 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdnnr\" (UniqueName: \"kubernetes.io/projected/05db8d30-1a63-44fc-af00-baac2f1825bf-kube-api-access-bdnnr\") pod \"limitador-limitador-54d6d988d5-jzq2l\" (UID: \"05db8d30-1a63-44fc-af00-baac2f1825bf\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jzq2l" Jul 08 03:08:28.069866 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:28.069803 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jzq2l" Jul 08 03:08:28.214118 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:28.214096 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:08:28.216588 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:08:28.216560 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05db8d30_1a63_44fc_af00_baac2f1825bf.slice/crio-5ee14663e5d62f0ab4ad5ab546e10e93b78831fcf567dafd0688d759444e0877 WatchSource:0}: Error finding container 5ee14663e5d62f0ab4ad5ab546e10e93b78831fcf567dafd0688d759444e0877: Status 404 returned error can't find the container with id 5ee14663e5d62f0ab4ad5ab546e10e93b78831fcf567dafd0688d759444e0877 Jul 08 03:08:28.535390 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:28.535351 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jzq2l" event={"ID":"05db8d30-1a63-44fc-af00-baac2f1825bf","Type":"ContainerStarted","Data":"5ee14663e5d62f0ab4ad5ab546e10e93b78831fcf567dafd0688d759444e0877"} Jul 08 03:08:31.550987 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:31.550951 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jzq2l" event={"ID":"05db8d30-1a63-44fc-af00-baac2f1825bf","Type":"ContainerStarted","Data":"ead7878e3e5d497fa7eb2b55fcf946ee530741477e0ee4928593e914065ec5e4"} Jul 08 03:08:31.551385 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:31.551009 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jzq2l" Jul 08 03:08:42.557724 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:42.557692 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jzq2l" Jul 08 03:08:42.577668 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:08:42.577615 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-jzq2l" podStartSLOduration=13.028611343 podStartE2EDuration="15.577597443s" totalImagesPullingTime="2.5489861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:08:31.569702312 +0000 UTC m=+630.560150349" watchObservedRunningTime="2026-07-08 03:08:42.577597443 +0000 UTC m=+641.568045481" Jul 08 03:09:42.712329 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:09:42.712290 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:09:58.524389 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:09:58.524297 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:10:21.906852 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:21.906809 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:10:28.728842 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.728807 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv"] Jul 08 03:10:28.732192 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.732173 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.736568 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.736543 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 03:10:28.736680 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.736567 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 08 03:10:28.736680 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.736568 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 03:10:28.736680 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.736552 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mv7gx\"" Jul 08 03:10:28.742613 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.742594 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv"] Jul 08 03:10:28.809582 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.809552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa37267-37b4-4518-97b1-9d8b3be28cb6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.809750 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.809587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa37267-37b4-4518-97b1-9d8b3be28cb6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.809750 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.809606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa37267-37b4-4518-97b1-9d8b3be28cb6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.809750 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.809689 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa37267-37b4-4518-97b1-9d8b3be28cb6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.809750 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.809725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa37267-37b4-4518-97b1-9d8b3be28cb6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.809911 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.809763 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8sdf\" (UniqueName: \"kubernetes.io/projected/afa37267-37b4-4518-97b1-9d8b3be28cb6-kube-api-access-m8sdf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.910833 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.910796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa37267-37b4-4518-97b1-9d8b3be28cb6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.910997 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.910840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8sdf\" (UniqueName: \"kubernetes.io/projected/afa37267-37b4-4518-97b1-9d8b3be28cb6-kube-api-access-m8sdf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.910997 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.910962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa37267-37b4-4518-97b1-9d8b3be28cb6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.911108 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.911008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa37267-37b4-4518-97b1-9d8b3be28cb6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.911108 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.911031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa37267-37b4-4518-97b1-9d8b3be28cb6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.911197 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.911147 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa37267-37b4-4518-97b1-9d8b3be28cb6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.911380 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.911362 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa37267-37b4-4518-97b1-9d8b3be28cb6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.911457 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.911439 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa37267-37b4-4518-97b1-9d8b3be28cb6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.911506 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.911471 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa37267-37b4-4518-97b1-9d8b3be28cb6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.913308 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.913289 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa37267-37b4-4518-97b1-9d8b3be28cb6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.913473 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.913456 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa37267-37b4-4518-97b1-9d8b3be28cb6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:28.919281 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:28.919253 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8sdf\" (UniqueName: \"kubernetes.io/projected/afa37267-37b4-4518-97b1-9d8b3be28cb6-kube-api-access-m8sdf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h5mqv\" (UID: \"afa37267-37b4-4518-97b1-9d8b3be28cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:29.042589 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:29.042503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:29.184196 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:29.184174 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv"] Jul 08 03:10:29.186545 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:10:29.186510 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa37267_37b4_4518_97b1_9d8b3be28cb6.slice/crio-81c4c24c5ad4df535a5f9949741f5bffe543a6ea4aea60ddeaa266f389d2c084 WatchSource:0}: Error finding container 81c4c24c5ad4df535a5f9949741f5bffe543a6ea4aea60ddeaa266f389d2c084: Status 404 returned error can't find the container with id 81c4c24c5ad4df535a5f9949741f5bffe543a6ea4aea60ddeaa266f389d2c084 Jul 08 03:10:30.112526 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:30.112486 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:10:30.131331 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:30.131292 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" event={"ID":"afa37267-37b4-4518-97b1-9d8b3be28cb6","Type":"ContainerStarted","Data":"81c4c24c5ad4df535a5f9949741f5bffe543a6ea4aea60ddeaa266f389d2c084"} Jul 08 03:10:31.137819 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:31.137782 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" event={"ID":"afa37267-37b4-4518-97b1-9d8b3be28cb6","Type":"ContainerStarted","Data":"ddc294cf97c131dd191a994978699ed5112757ebc5a15da093fe7d2cd7ae246b"} Jul 08 03:10:31.156212 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:31.156118 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" podStartSLOduration=1.944252069 podStartE2EDuration="3.156096741s" totalImagesPullingTime="1.211844672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:10:31.155441441 +0000 UTC m=+750.145889490" watchObservedRunningTime="2026-07-08 03:10:31.156096741 +0000 UTC m=+750.146544780" Jul 08 03:10:39.043475 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:39.043437 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:39.043929 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:39.043494 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:39.056087 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:39.056036 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:39.188663 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:39.188631 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h5mqv" Jul 08 03:10:42.909888 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:10:42.909855 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:11:06.322416 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:11:06.322382 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:11:20.016737 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:11:20.016704 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:12:44.618826 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:12:44.618790 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:12:54.713099 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:12:54.713000 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:13:01.494198 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:13:01.494171 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:13:01.495533 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:13:01.495507 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:13:01.500213 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:13:01.500188 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:13:01.501470 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:13:01.501445 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:13:04.115776 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:13:04.115732 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:13:13.614424 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:13:13.614385 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:13:24.609512 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:13:24.609467 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:13:51.233449 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:13:51.233410 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:14:03.313123 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:14:03.313086 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:14:37.119343 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:14:37.119266 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:14:46.620321 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:14:46.620282 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:14:56.015058 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:14:56.015005 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:15:00.155029 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:00.154995 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm"] Jul 08 03:15:00.158445 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:00.158424 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm" Jul 08 03:15:00.161568 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:00.161545 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 03:15:00.162775 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:00.162744 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 03:15:00.162775 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:00.162766 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jdwqj\"" Jul 08 03:15:00.168477 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:00.168453 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm"] Jul 08 03:15:00.285278 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:00.285243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntfg2\" (UniqueName: \"kubernetes.io/projected/4670ce9b-ff4e-4ec5-a356-f6d553b89fa5-kube-api-access-ntfg2\") pod \"maas-api-key-cleanup-29724675-mslcm\" (UID: \"4670ce9b-ff4e-4ec5-a356-f6d553b89fa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm" Jul 08 03:15:00.385850 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:00.385809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntfg2\" (UniqueName: \"kubernetes.io/projected/4670ce9b-ff4e-4ec5-a356-f6d553b89fa5-kube-api-access-ntfg2\") pod \"maas-api-key-cleanup-29724675-mslcm\" (UID: \"4670ce9b-ff4e-4ec5-a356-f6d553b89fa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm" Jul 08 03:15:00.395426 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:00.395395 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntfg2\" (UniqueName: \"kubernetes.io/projected/4670ce9b-ff4e-4ec5-a356-f6d553b89fa5-kube-api-access-ntfg2\") pod \"maas-api-key-cleanup-29724675-mslcm\" (UID: \"4670ce9b-ff4e-4ec5-a356-f6d553b89fa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm" Jul 08 03:15:00.469909 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:00.469883 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm" Jul 08 03:15:00.609731 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:00.609708 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm"] Jul 08 03:15:00.611887 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:15:00.611857 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4670ce9b_ff4e_4ec5_a356_f6d553b89fa5.slice/crio-bd3dc001d1b3aa268537d5794e7ae5639c7562d264d9c00c228231f5f2f33cbb WatchSource:0}: Error finding container bd3dc001d1b3aa268537d5794e7ae5639c7562d264d9c00c228231f5f2f33cbb: Status 404 returned error can't find the container with id bd3dc001d1b3aa268537d5794e7ae5639c7562d264d9c00c228231f5f2f33cbb Jul 08 03:15:00.613611 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:00.613594 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 03:15:01.495416 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:01.495382 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm" event={"ID":"4670ce9b-ff4e-4ec5-a356-f6d553b89fa5","Type":"ContainerStarted","Data":"bd3dc001d1b3aa268537d5794e7ae5639c7562d264d9c00c228231f5f2f33cbb"} Jul 08 03:15:03.507487 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:03.507399 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm" event={"ID":"4670ce9b-ff4e-4ec5-a356-f6d553b89fa5","Type":"ContainerStarted","Data":"480477e3df5e56bc5b7180e7b8b4fe73934eef07ff48203301b5177a799a02b4"} Jul 08 03:15:03.526565 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:03.526516 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm" podStartSLOduration=1.24170999 podStartE2EDuration="3.526501203s" totalImagesPullingTime="2.284791213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:15:03.524029963 +0000 UTC m=+1022.514478000" watchObservedRunningTime="2026-07-08 03:15:03.526501203 +0000 UTC m=+1022.516949239" Jul 08 03:15:06.717101 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:06.717040 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:15:16.116539 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:16.116503 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:15:27.222377 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:27.222342 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:15:38.814502 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:38.814459 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:15:49.013919 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:15:49.013882 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:16:00.010522 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:00.010437 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm"] Jul 08 03:16:00.010958 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:00.010678 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm" podUID="4670ce9b-ff4e-4ec5-a356-f6d553b89fa5" containerName="cleanup" containerID="cri-o://480477e3df5e56bc5b7180e7b8b4fe73934eef07ff48203301b5177a799a02b4" gracePeriod=30 Jul 08 03:16:11.121935 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:11.121893 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:16:20.806274 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:20.806243 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:16:30.204517 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:30.204490 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm" Jul 08 03:16:30.257762 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:30.257732 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntfg2\" (UniqueName: \"kubernetes.io/projected/4670ce9b-ff4e-4ec5-a356-f6d553b89fa5-kube-api-access-ntfg2\") pod \"4670ce9b-ff4e-4ec5-a356-f6d553b89fa5\" (UID: \"4670ce9b-ff4e-4ec5-a356-f6d553b89fa5\") " Jul 08 03:16:30.259991 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:30.259963 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4670ce9b-ff4e-4ec5-a356-f6d553b89fa5-kube-api-access-ntfg2" (OuterVolumeSpecName: "kube-api-access-ntfg2") pod "4670ce9b-ff4e-4ec5-a356-f6d553b89fa5" (UID: "4670ce9b-ff4e-4ec5-a356-f6d553b89fa5"). InnerVolumeSpecName "kube-api-access-ntfg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:16:30.358458 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:30.358418 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntfg2\" (UniqueName: \"kubernetes.io/projected/4670ce9b-ff4e-4ec5-a356-f6d553b89fa5-kube-api-access-ntfg2\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:16:30.936279 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:30.936247 2576 generic.go:356] "Generic (PLEG): container finished" podID="4670ce9b-ff4e-4ec5-a356-f6d553b89fa5" containerID="480477e3df5e56bc5b7180e7b8b4fe73934eef07ff48203301b5177a799a02b4" exitCode=137 Jul 08 03:16:30.936466 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:30.936293 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm" Jul 08 03:16:30.936466 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:30.936312 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm" event={"ID":"4670ce9b-ff4e-4ec5-a356-f6d553b89fa5","Type":"ContainerDied","Data":"480477e3df5e56bc5b7180e7b8b4fe73934eef07ff48203301b5177a799a02b4"} Jul 08 03:16:30.936466 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:30.936354 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm" event={"ID":"4670ce9b-ff4e-4ec5-a356-f6d553b89fa5","Type":"ContainerDied","Data":"bd3dc001d1b3aa268537d5794e7ae5639c7562d264d9c00c228231f5f2f33cbb"} Jul 08 03:16:30.936466 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:30.936374 2576 scope.go:117] "RemoveContainer" containerID="480477e3df5e56bc5b7180e7b8b4fe73934eef07ff48203301b5177a799a02b4" Jul 08 03:16:30.945856 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:30.945840 2576 scope.go:117] "RemoveContainer" containerID="480477e3df5e56bc5b7180e7b8b4fe73934eef07ff48203301b5177a799a02b4" Jul 08 03:16:30.946116 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:16:30.946096 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480477e3df5e56bc5b7180e7b8b4fe73934eef07ff48203301b5177a799a02b4\": container with ID starting with 480477e3df5e56bc5b7180e7b8b4fe73934eef07ff48203301b5177a799a02b4 not found: ID does not exist" containerID="480477e3df5e56bc5b7180e7b8b4fe73934eef07ff48203301b5177a799a02b4" Jul 08 03:16:30.946193 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:30.946124 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480477e3df5e56bc5b7180e7b8b4fe73934eef07ff48203301b5177a799a02b4"} err="failed to get container status \"480477e3df5e56bc5b7180e7b8b4fe73934eef07ff48203301b5177a799a02b4\": rpc error: code = NotFound desc = could not find container \"480477e3df5e56bc5b7180e7b8b4fe73934eef07ff48203301b5177a799a02b4\": container with ID starting with 480477e3df5e56bc5b7180e7b8b4fe73934eef07ff48203301b5177a799a02b4 not found: ID does not exist" Jul 08 03:16:30.958743 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:30.958714 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm"] Jul 08 03:16:30.961791 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:30.961768 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724675-mslcm"] Jul 08 03:16:31.527278 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:31.527234 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4670ce9b-ff4e-4ec5-a356-f6d553b89fa5" path="/var/lib/kubelet/pods/4670ce9b-ff4e-4ec5-a356-f6d553b89fa5/volumes" Jul 08 03:16:44.620579 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:44.620535 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:16:54.919070 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:16:54.919006 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:17:10.209602 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:17:10.209562 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:17:19.616040 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:17:19.616003 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:17:31.421079 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:17:31.420984 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:17:43.318535 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:17:43.318497 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:18:01.527718 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:01.527689 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:18:01.528766 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:01.528745 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:18:01.533128 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:01.533105 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:18:01.534034 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:01.534017 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:18:14.331673 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:14.331644 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29"] Jul 08 03:18:14.332253 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:14.331847 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" podUID="383c043f-be58-435d-a239-26a03c31b9cb" containerName="manager" containerID="cri-o://3271a85b5abb0915bad2f2e0d6201c175cd32937eb9ed76d05c57a5b577db8cc" gracePeriod=10 Jul 08 03:18:17.298990 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.298964 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" Jul 08 03:18:17.402934 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.402852 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/383c043f-be58-435d-a239-26a03c31b9cb-extensions-socket-volume\") pod \"383c043f-be58-435d-a239-26a03c31b9cb\" (UID: \"383c043f-be58-435d-a239-26a03c31b9cb\") " Jul 08 03:18:17.403098 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.402938 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgr9n\" (UniqueName: \"kubernetes.io/projected/383c043f-be58-435d-a239-26a03c31b9cb-kube-api-access-tgr9n\") pod \"383c043f-be58-435d-a239-26a03c31b9cb\" (UID: \"383c043f-be58-435d-a239-26a03c31b9cb\") " Jul 08 03:18:17.403255 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.403218 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383c043f-be58-435d-a239-26a03c31b9cb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "383c043f-be58-435d-a239-26a03c31b9cb" (UID: "383c043f-be58-435d-a239-26a03c31b9cb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:18:17.405036 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.405002 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383c043f-be58-435d-a239-26a03c31b9cb-kube-api-access-tgr9n" (OuterVolumeSpecName: "kube-api-access-tgr9n") pod "383c043f-be58-435d-a239-26a03c31b9cb" (UID: "383c043f-be58-435d-a239-26a03c31b9cb"). InnerVolumeSpecName "kube-api-access-tgr9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:18:17.470990 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.470961 2576 generic.go:356] "Generic (PLEG): container finished" podID="383c043f-be58-435d-a239-26a03c31b9cb" containerID="3271a85b5abb0915bad2f2e0d6201c175cd32937eb9ed76d05c57a5b577db8cc" exitCode=0 Jul 08 03:18:17.471140 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.471005 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" Jul 08 03:18:17.471140 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.471074 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" event={"ID":"383c043f-be58-435d-a239-26a03c31b9cb","Type":"ContainerDied","Data":"3271a85b5abb0915bad2f2e0d6201c175cd32937eb9ed76d05c57a5b577db8cc"} Jul 08 03:18:17.471221 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.471159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29" event={"ID":"383c043f-be58-435d-a239-26a03c31b9cb","Type":"ContainerDied","Data":"ff4638edc770bcdd57a502307f90758636323f24cb92f97ef3d0a8a36eaa9779"} Jul 08 03:18:17.471221 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.471188 2576 scope.go:117] "RemoveContainer" containerID="3271a85b5abb0915bad2f2e0d6201c175cd32937eb9ed76d05c57a5b577db8cc" Jul 08 03:18:17.480997 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.480982 2576 scope.go:117] "RemoveContainer" containerID="3271a85b5abb0915bad2f2e0d6201c175cd32937eb9ed76d05c57a5b577db8cc" Jul 08 03:18:17.481285 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:18:17.481264 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3271a85b5abb0915bad2f2e0d6201c175cd32937eb9ed76d05c57a5b577db8cc\": container with ID starting with 3271a85b5abb0915bad2f2e0d6201c175cd32937eb9ed76d05c57a5b577db8cc not found: ID does not exist" containerID="3271a85b5abb0915bad2f2e0d6201c175cd32937eb9ed76d05c57a5b577db8cc" Jul 08 03:18:17.481372 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.481292 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3271a85b5abb0915bad2f2e0d6201c175cd32937eb9ed76d05c57a5b577db8cc"} err="failed to get container status \"3271a85b5abb0915bad2f2e0d6201c175cd32937eb9ed76d05c57a5b577db8cc\": rpc error: code = NotFound desc = could not find container \"3271a85b5abb0915bad2f2e0d6201c175cd32937eb9ed76d05c57a5b577db8cc\": container with ID starting with 3271a85b5abb0915bad2f2e0d6201c175cd32937eb9ed76d05c57a5b577db8cc not found: ID does not exist" Jul 08 03:18:17.491418 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.491399 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29"] Jul 08 03:18:17.495100 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.495081 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8df29"] Jul 08 03:18:17.504594 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.504576 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/383c043f-be58-435d-a239-26a03c31b9cb-extensions-socket-volume\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:18:17.504670 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.504596 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgr9n\" (UniqueName: \"kubernetes.io/projected/383c043f-be58-435d-a239-26a03c31b9cb-kube-api-access-tgr9n\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:18:17.525819 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:18:17.525798 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="383c043f-be58-435d-a239-26a03c31b9cb" path="/var/lib/kubelet/pods/383c043f-be58-435d-a239-26a03c31b9cb/volumes" Jul 08 03:19:15.460422 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.460385 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd"] Jul 08 03:19:15.462967 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.460907 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383c043f-be58-435d-a239-26a03c31b9cb" containerName="manager" Jul 08 03:19:15.462967 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.460924 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="383c043f-be58-435d-a239-26a03c31b9cb" containerName="manager" Jul 08 03:19:15.462967 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.460933 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4670ce9b-ff4e-4ec5-a356-f6d553b89fa5" containerName="cleanup" Jul 08 03:19:15.462967 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.460938 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4670ce9b-ff4e-4ec5-a356-f6d553b89fa5" containerName="cleanup" Jul 08 03:19:15.462967 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.461033 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="383c043f-be58-435d-a239-26a03c31b9cb" containerName="manager" Jul 08 03:19:15.462967 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.461062 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4670ce9b-ff4e-4ec5-a356-f6d553b89fa5" containerName="cleanup" Jul 08 03:19:15.464111 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.464095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd" Jul 08 03:19:15.466869 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.466848 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8bn97\"" Jul 08 03:19:15.478341 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.478315 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd"] Jul 08 03:19:15.595123 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.595092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07a3cb3f-6867-4fba-b0c5-9f3294233ca6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dfxcd\" (UID: \"07a3cb3f-6867-4fba-b0c5-9f3294233ca6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd" Jul 08 03:19:15.595299 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.595271 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjgs5\" (UniqueName: \"kubernetes.io/projected/07a3cb3f-6867-4fba-b0c5-9f3294233ca6-kube-api-access-sjgs5\") pod \"kuadrant-operator-controller-manager-86878c769f-dfxcd\" (UID: \"07a3cb3f-6867-4fba-b0c5-9f3294233ca6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd" Jul 08 03:19:15.696205 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.696174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjgs5\" (UniqueName: \"kubernetes.io/projected/07a3cb3f-6867-4fba-b0c5-9f3294233ca6-kube-api-access-sjgs5\") pod \"kuadrant-operator-controller-manager-86878c769f-dfxcd\" (UID: \"07a3cb3f-6867-4fba-b0c5-9f3294233ca6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd" Jul 08 03:19:15.696367 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.696223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07a3cb3f-6867-4fba-b0c5-9f3294233ca6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dfxcd\" (UID: \"07a3cb3f-6867-4fba-b0c5-9f3294233ca6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd" Jul 08 03:19:15.696614 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.696597 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07a3cb3f-6867-4fba-b0c5-9f3294233ca6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dfxcd\" (UID: \"07a3cb3f-6867-4fba-b0c5-9f3294233ca6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd" Jul 08 03:19:15.705656 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.705622 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjgs5\" (UniqueName: \"kubernetes.io/projected/07a3cb3f-6867-4fba-b0c5-9f3294233ca6-kube-api-access-sjgs5\") pod \"kuadrant-operator-controller-manager-86878c769f-dfxcd\" (UID: \"07a3cb3f-6867-4fba-b0c5-9f3294233ca6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd" Jul 08 03:19:15.774725 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.774638 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd" Jul 08 03:19:15.924202 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:15.924117 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd"] Jul 08 03:19:15.926936 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:19:15.926906 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a3cb3f_6867_4fba_b0c5_9f3294233ca6.slice/crio-8ae6a98734dd0a8715968dd5ab70fc2953d3a0df67c58873dcdf22e8420f4ea0 WatchSource:0}: Error finding container 8ae6a98734dd0a8715968dd5ab70fc2953d3a0df67c58873dcdf22e8420f4ea0: Status 404 returned error can't find the container with id 8ae6a98734dd0a8715968dd5ab70fc2953d3a0df67c58873dcdf22e8420f4ea0 Jul 08 03:19:16.764484 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:16.764444 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd" event={"ID":"07a3cb3f-6867-4fba-b0c5-9f3294233ca6","Type":"ContainerStarted","Data":"ae881bc5ff972c20b4f1290c80afeaec77bdbe1b0bc6f070715e260d4cdb951a"} Jul 08 03:19:16.764888 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:16.764497 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd" event={"ID":"07a3cb3f-6867-4fba-b0c5-9f3294233ca6","Type":"ContainerStarted","Data":"8ae6a98734dd0a8715968dd5ab70fc2953d3a0df67c58873dcdf22e8420f4ea0"} Jul 08 03:19:16.764888 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:16.764539 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd" Jul 08 03:19:16.787377 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:16.787322 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd" podStartSLOduration=1.787304803 podStartE2EDuration="1.787304803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:19:16.784508223 +0000 UTC m=+1275.774956251" watchObservedRunningTime="2026-07-08 03:19:16.787304803 +0000 UTC m=+1275.777752839" Jul 08 03:19:27.771483 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:27.771453 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfxcd" Jul 08 03:19:37.315981 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:37.315942 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:19:46.117133 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:46.117096 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:19:59.213236 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:19:59.213194 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:20:10.413541 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:20:10.413506 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:20:21.420098 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:20:21.420057 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:20:32.313482 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:20:32.313249 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:20:41.412787 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:20:41.412755 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:20:52.316058 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:20:52.316012 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:21:02.021095 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:21:02.021059 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:21:12.716219 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:21:12.716177 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:21:25.103964 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:21:25.103926 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:21:46.713959 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:21:46.713920 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:21:56.112675 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:21:56.112593 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:22:06.207557 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:22:06.207523 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:22:16.007371 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:22:16.007335 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:22:33.510506 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:22:33.510469 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:22:42.708103 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:22:42.708064 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:22:51.821019 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:22:51.820985 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:23:01.556999 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:23:01.556968 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:23:01.560111 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:23:01.560088 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:23:01.563353 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:23:01.563332 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:23:01.566392 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:23:01.566372 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:23:01.915562 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:23:01.915533 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:23:11.119987 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:23:11.119948 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:23:20.909663 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:23:20.909622 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:23:34.813871 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:23:34.813780 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:23:48.715421 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:23:48.715383 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:23:58.016436 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:23:58.016399 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:24:07.126247 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:24:07.126209 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:24:19.112919 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:24:19.112874 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:24:36.115365 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:24:36.115329 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:24:47.229637 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:24:47.229600 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:24:56.416306 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:24:56.416220 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:25:08.709836 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:25:08.709799 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:25:17.618792 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:25:17.618743 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:25:27.128867 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:25:27.128837 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:25:46.414264 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:25:46.414232 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:25:56.413772 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:25:56.413733 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:26:04.119483 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:26:04.119452 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:26:12.822116 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:26:12.822080 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:28:01.590121 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:28:01.590093 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:28:01.593396 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:28:01.593369 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:28:01.595468 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:28:01.595447 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:28:01.599037 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:28:01.599021 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:28:11.915382 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:28:11.915342 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:28:19.610485 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:28:19.610453 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:29:37.127979 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:29:37.127891 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:29:54.408134 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:29:54.408030 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:30:00.138530 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:00.138500 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8"] Jul 08 03:30:00.142185 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:00.142170 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8" Jul 08 03:30:00.144818 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:00.144795 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 03:30:00.145992 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:00.145971 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jdwqj\"" Jul 08 03:30:00.146128 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:00.145974 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 03:30:00.149545 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:00.149524 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8"] Jul 08 03:30:00.203549 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:00.203514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drnl7\" (UniqueName: \"kubernetes.io/projected/99fa019d-70bf-4430-8169-41bb9d512681-kube-api-access-drnl7\") pod \"maas-api-key-cleanup-29724690-dglp8\" (UID: \"99fa019d-70bf-4430-8169-41bb9d512681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8" Jul 08 03:30:00.304295 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:00.304260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drnl7\" (UniqueName: \"kubernetes.io/projected/99fa019d-70bf-4430-8169-41bb9d512681-kube-api-access-drnl7\") pod \"maas-api-key-cleanup-29724690-dglp8\" (UID: \"99fa019d-70bf-4430-8169-41bb9d512681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8" Jul 08 03:30:00.313585 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:00.313560 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drnl7\" (UniqueName: \"kubernetes.io/projected/99fa019d-70bf-4430-8169-41bb9d512681-kube-api-access-drnl7\") pod \"maas-api-key-cleanup-29724690-dglp8\" (UID: \"99fa019d-70bf-4430-8169-41bb9d512681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8" Jul 08 03:30:00.453724 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:00.453672 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8" Jul 08 03:30:00.593959 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:00.593896 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8"] Jul 08 03:30:00.596649 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:30:00.596621 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99fa019d_70bf_4430_8169_41bb9d512681.slice/crio-15157efb97a3d11a6f1706d2b648ef5adee1aa18201809396187516a79d11827 WatchSource:0}: Error finding container 15157efb97a3d11a6f1706d2b648ef5adee1aa18201809396187516a79d11827: Status 404 returned error can't find the container with id 15157efb97a3d11a6f1706d2b648ef5adee1aa18201809396187516a79d11827 Jul 08 03:30:00.989942 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:00.989901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8" event={"ID":"99fa019d-70bf-4430-8169-41bb9d512681","Type":"ContainerStarted","Data":"0fa126763b9af90d5639dc492e94e74d70b9dc6b01aafc999af440a76826d982"} Jul 08 03:30:00.990145 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:00.989967 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8" event={"ID":"99fa019d-70bf-4430-8169-41bb9d512681","Type":"ContainerStarted","Data":"15157efb97a3d11a6f1706d2b648ef5adee1aa18201809396187516a79d11827"} Jul 08 03:30:01.007348 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:01.007298 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8" podStartSLOduration=1.007284783 podStartE2EDuration="1.007284783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:30:01.005707594 +0000 UTC m=+1919.996155653" watchObservedRunningTime="2026-07-08 03:30:01.007284783 +0000 UTC m=+1919.997732871" Jul 08 03:30:02.707904 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:02.707869 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:30:31.612246 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:31.612207 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:30:56.610065 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:30:56.610005 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:31:00.011193 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:00.011158 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8"] Jul 08 03:31:00.011593 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:00.011373 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8" podUID="99fa019d-70bf-4430-8169-41bb9d512681" containerName="cleanup" containerID="cri-o://0fa126763b9af90d5639dc492e94e74d70b9dc6b01aafc999af440a76826d982" gracePeriod=30 Jul 08 03:31:23.214901 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:23.214871 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:31:30.202699 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:30.202669 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8" Jul 08 03:31:30.283297 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:30.283208 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drnl7\" (UniqueName: \"kubernetes.io/projected/99fa019d-70bf-4430-8169-41bb9d512681-kube-api-access-drnl7\") pod \"99fa019d-70bf-4430-8169-41bb9d512681\" (UID: \"99fa019d-70bf-4430-8169-41bb9d512681\") " Jul 08 03:31:30.285383 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:30.285357 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99fa019d-70bf-4430-8169-41bb9d512681-kube-api-access-drnl7" (OuterVolumeSpecName: "kube-api-access-drnl7") pod "99fa019d-70bf-4430-8169-41bb9d512681" (UID: "99fa019d-70bf-4430-8169-41bb9d512681"). InnerVolumeSpecName "kube-api-access-drnl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:31:30.384749 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:30.384717 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drnl7\" (UniqueName: \"kubernetes.io/projected/99fa019d-70bf-4430-8169-41bb9d512681-kube-api-access-drnl7\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:31:30.440696 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:30.440660 2576 generic.go:356] "Generic (PLEG): container finished" podID="99fa019d-70bf-4430-8169-41bb9d512681" containerID="0fa126763b9af90d5639dc492e94e74d70b9dc6b01aafc999af440a76826d982" exitCode=137 Jul 08 03:31:30.440842 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:30.440709 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8" Jul 08 03:31:30.440842 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:30.440754 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8" event={"ID":"99fa019d-70bf-4430-8169-41bb9d512681","Type":"ContainerDied","Data":"0fa126763b9af90d5639dc492e94e74d70b9dc6b01aafc999af440a76826d982"} Jul 08 03:31:30.440842 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:30.440811 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8" event={"ID":"99fa019d-70bf-4430-8169-41bb9d512681","Type":"ContainerDied","Data":"15157efb97a3d11a6f1706d2b648ef5adee1aa18201809396187516a79d11827"} Jul 08 03:31:30.440943 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:30.440841 2576 scope.go:117] "RemoveContainer" containerID="0fa126763b9af90d5639dc492e94e74d70b9dc6b01aafc999af440a76826d982" Jul 08 03:31:30.450014 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:30.449995 2576 scope.go:117] "RemoveContainer" containerID="0fa126763b9af90d5639dc492e94e74d70b9dc6b01aafc999af440a76826d982" Jul 08 03:31:30.450333 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:31:30.450304 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa126763b9af90d5639dc492e94e74d70b9dc6b01aafc999af440a76826d982\": container with ID starting with 0fa126763b9af90d5639dc492e94e74d70b9dc6b01aafc999af440a76826d982 not found: ID does not exist" containerID="0fa126763b9af90d5639dc492e94e74d70b9dc6b01aafc999af440a76826d982" Jul 08 03:31:30.450410 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:30.450345 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa126763b9af90d5639dc492e94e74d70b9dc6b01aafc999af440a76826d982"} err="failed to get container status \"0fa126763b9af90d5639dc492e94e74d70b9dc6b01aafc999af440a76826d982\": rpc error: code = NotFound desc = could not find container \"0fa126763b9af90d5639dc492e94e74d70b9dc6b01aafc999af440a76826d982\": container with ID starting with 0fa126763b9af90d5639dc492e94e74d70b9dc6b01aafc999af440a76826d982 not found: ID does not exist" Jul 08 03:31:30.464032 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:30.463995 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8"] Jul 08 03:31:30.466203 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:30.466181 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724690-dglp8"] Jul 08 03:31:31.526142 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:31.526111 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99fa019d-70bf-4430-8169-41bb9d512681" path="/var/lib/kubelet/pods/99fa019d-70bf-4430-8169-41bb9d512681/volumes" Jul 08 03:31:36.506978 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:31:36.506945 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:32:16.107994 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:32:16.107954 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:32:30.512946 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:32:30.512864 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:32:41.616392 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:32:41.616356 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:33:01.620275 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:33:01.620246 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:33:01.625227 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:33:01.625203 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:33:01.625959 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:33:01.625934 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:33:01.630557 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:33:01.630537 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:34:04.510240 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:04.510160 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:34:47.758261 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.758227 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7"] Jul 08 03:34:47.760865 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.759277 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99fa019d-70bf-4430-8169-41bb9d512681" containerName="cleanup" Jul 08 03:34:47.760865 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.759300 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="99fa019d-70bf-4430-8169-41bb9d512681" containerName="cleanup" Jul 08 03:34:47.760865 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.759434 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="99fa019d-70bf-4430-8169-41bb9d512681" containerName="cleanup" Jul 08 03:34:47.762526 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.762506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.765741 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.765705 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7142f2-openshift-default-dockercfg-pd5wx\"" Jul 08 03:34:47.765853 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.765747 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 03:34:47.778926 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.778893 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7"] Jul 08 03:34:47.805718 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.805687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-data\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.805849 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.805758 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-workload-certs\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.805849 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.805804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjfb\" (UniqueName: \"kubernetes.io/projected/984c5953-7506-4409-b2dd-4a0ffa7293fe-kube-api-access-4zjfb\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.805849 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.805832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-token\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.806034 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.805919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-workload-socket\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.806034 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.805959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-credential-socket\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.806034 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.805987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-envoy\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.806199 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.806088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984c5953-7506-4409-b2dd-4a0ffa7293fe-istiod-ca-cert\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.806199 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.806130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-podinfo\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.907225 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.907169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-workload-socket\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.907225 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.907222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-credential-socket\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.907225 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.907239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-envoy\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.907566 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.907281 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984c5953-7506-4409-b2dd-4a0ffa7293fe-istiod-ca-cert\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.907566 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.907303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-podinfo\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.907566 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.907330 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-data\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.907566 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.907382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-workload-certs\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.907566 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.907434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjfb\" (UniqueName: \"kubernetes.io/projected/984c5953-7506-4409-b2dd-4a0ffa7293fe-kube-api-access-4zjfb\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.907837 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.907772 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-credential-socket\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.907895 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.907840 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-data\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.907959 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.907890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-token\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.908031 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.908007 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-workload-certs\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.908121 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.908093 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984c5953-7506-4409-b2dd-4a0ffa7293fe-istiod-ca-cert\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.908368 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.908347 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-workload-socket\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.909818 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.909786 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-envoy\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.910008 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.909990 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-podinfo\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.919883 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.919858 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjfb\" (UniqueName: \"kubernetes.io/projected/984c5953-7506-4409-b2dd-4a0ffa7293fe-kube-api-access-4zjfb\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:47.920988 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:47.920963 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-token\") pod \"e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:48.083555 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:48.083466 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:48.233628 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:48.233424 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7"] Jul 08 03:34:48.235992 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:34:48.235967 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984c5953_7506_4409_b2dd_4a0ffa7293fe.slice/crio-ad4e3f4fd219a3ad9d907208b29438862b7a2a7e8ec2e98e287414966c09850c WatchSource:0}: Error finding container ad4e3f4fd219a3ad9d907208b29438862b7a2a7e8ec2e98e287414966c09850c: Status 404 returned error can't find the container with id ad4e3f4fd219a3ad9d907208b29438862b7a2a7e8ec2e98e287414966c09850c Jul 08 03:34:48.238055 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:48.238022 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 03:34:48.430442 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:48.430390 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" event={"ID":"984c5953-7506-4409-b2dd-4a0ffa7293fe","Type":"ContainerStarted","Data":"ad4e3f4fd219a3ad9d907208b29438862b7a2a7e8ec2e98e287414966c09850c"} Jul 08 03:34:50.221817 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:50.221779 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:34:51.937871 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:51.937834 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:34:51.938161 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:51.937911 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:34:51.938161 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:51.937945 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:34:52.453358 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:52.453316 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" event={"ID":"984c5953-7506-4409-b2dd-4a0ffa7293fe","Type":"ContainerStarted","Data":"6e621d53b3395a5adc9ee906e8bc4966bfcac1c963d57fe8d4dac7d2fd8a8816"} Jul 08 03:34:52.476524 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:52.476477 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" podStartSLOduration=1.777177058 podStartE2EDuration="5.476461656s" totalImagesPullingTime="3.699284598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:34:52.47342308 +0000 UTC m=+2211.463871117" watchObservedRunningTime="2026-07-08 03:34:52.476461656 +0000 UTC m=+2211.466909689" Jul 08 03:34:53.084122 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:53.084090 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:53.089023 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:53.089000 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:53.458989 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:53.458955 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:34:53.459834 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:34:53.459813 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:35:06.526795 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:06.526759 2576 generic.go:356] "Generic (PLEG): container finished" podID="984c5953-7506-4409-b2dd-4a0ffa7293fe" containerID="6e621d53b3395a5adc9ee906e8bc4966bfcac1c963d57fe8d4dac7d2fd8a8816" exitCode=0 Jul 08 03:35:06.527223 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:06.526832 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" event={"ID":"984c5953-7506-4409-b2dd-4a0ffa7293fe","Type":"ContainerDied","Data":"6e621d53b3395a5adc9ee906e8bc4966bfcac1c963d57fe8d4dac7d2fd8a8816"} Jul 08 03:35:06.527266 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:06.527232 2576 scope.go:117] "RemoveContainer" containerID="6e621d53b3395a5adc9ee906e8bc4966bfcac1c963d57fe8d4dac7d2fd8a8816" Jul 08 03:35:06.528256 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:06.528228 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:35:06.528337 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:06.528291 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:35:06.528337 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:06.528321 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:35:07.020502 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:07.020460 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:35:07.533853 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:07.533820 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" event={"ID":"984c5953-7506-4409-b2dd-4a0ffa7293fe","Type":"ContainerStarted","Data":"ae5e18a7128be5a8d403eb67d1a1e96ae1df970c557abf1f06fe783a5c64fc9a"} Jul 08 03:35:08.085468 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:08.085427 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:35:08.085779 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:08.085553 2576 patch_prober.go:28] interesting pod/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 08 03:35:08.085779 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:08.085611 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" podUID="984c5953-7506-4409-b2dd-4a0ffa7293fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 08 03:35:08.459331 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:08.459289 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:35:09.084089 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:09.084030 2576 patch_prober.go:28] interesting pod/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 08 03:35:09.084482 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:09.084109 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" podUID="984c5953-7506-4409-b2dd-4a0ffa7293fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 08 03:35:10.084764 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:10.084698 2576 patch_prober.go:28] interesting pod/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 08 03:35:10.085181 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:10.084815 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" podUID="984c5953-7506-4409-b2dd-4a0ffa7293fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 08 03:35:11.084504 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:11.084471 2576 patch_prober.go:28] interesting pod/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 08 03:35:11.084679 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:11.084534 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" podUID="984c5953-7506-4409-b2dd-4a0ffa7293fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 08 03:35:12.089805 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:12.089776 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:35:12.091152 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:12.091121 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:35:16.118388 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:16.118354 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:35:27.002735 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:27.002649 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:35:42.227233 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:42.227192 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:35:56.022988 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:56.022951 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:35:57.081559 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.081528 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2"] Jul 08 03:35:57.085828 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.085806 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.088649 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.088627 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-reuse-25332d-openshift-default-dockercfg-5lxlx\"" Jul 08 03:35:57.096936 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.096912 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2"] Jul 08 03:35:57.126792 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.126761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-token\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.126905 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.126807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-workload-socket\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.126905 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.126843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-workload-certs\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.126905 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.126874 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-podinfo\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.127035 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.126906 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d84118-0aa0-4256-9fdb-8c996dd02365-istiod-ca-cert\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.127035 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.126938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-envoy\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.127141 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.127028 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-credential-socket\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.127141 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.127074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-data\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.127141 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.127096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkwlg\" (UniqueName: \"kubernetes.io/projected/95d84118-0aa0-4256-9fdb-8c996dd02365-kube-api-access-nkwlg\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.227757 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.227724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-workload-socket\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.227757 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.227764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-workload-certs\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.228020 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.227785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-podinfo\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.228020 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.227803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d84118-0aa0-4256-9fdb-8c996dd02365-istiod-ca-cert\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.228020 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.227931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-envoy\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.228216 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.228024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-credential-socket\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.228216 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.228105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-data\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.228216 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.228132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkwlg\" (UniqueName: \"kubernetes.io/projected/95d84118-0aa0-4256-9fdb-8c996dd02365-kube-api-access-nkwlg\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.228216 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.228179 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-workload-certs\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.228216 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.228203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-token\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.228479 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.228371 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-workload-socket\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.228549 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.228515 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-data\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.228630 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.228552 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-credential-socket\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.228692 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.228635 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d84118-0aa0-4256-9fdb-8c996dd02365-istiod-ca-cert\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.230554 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.230534 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-podinfo\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.230817 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.230801 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-envoy\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.237136 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.237099 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-token\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.237260 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.237245 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkwlg\" (UniqueName: \"kubernetes.io/projected/95d84118-0aa0-4256-9fdb-8c996dd02365-kube-api-access-nkwlg\") pod \"e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.397978 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.397896 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:57.556573 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.556549 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2"] Jul 08 03:35:57.558910 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:35:57.558876 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d84118_0aa0_4256_9fdb_8c996dd02365.slice/crio-6c6d644a8019670f89fff1fd5cc2324ce10699e1a63cece37f086a6c6b21c9f8 WatchSource:0}: Error finding container 6c6d644a8019670f89fff1fd5cc2324ce10699e1a63cece37f086a6c6b21c9f8: Status 404 returned error can't find the container with id 6c6d644a8019670f89fff1fd5cc2324ce10699e1a63cece37f086a6c6b21c9f8 Jul 08 03:35:57.561368 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.561334 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:35:57.561470 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.561401 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:35:57.561470 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.561430 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:35:57.787753 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.787716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" event={"ID":"95d84118-0aa0-4256-9fdb-8c996dd02365","Type":"ContainerStarted","Data":"72487042debd2eadd4c6f3693871f6646814f9501d09215098eb19dab3d2eb89"} Jul 08 03:35:57.787940 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.787781 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" event={"ID":"95d84118-0aa0-4256-9fdb-8c996dd02365","Type":"ContainerStarted","Data":"6c6d644a8019670f89fff1fd5cc2324ce10699e1a63cece37f086a6c6b21c9f8"} Jul 08 03:35:57.816709 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:57.816662 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" podStartSLOduration=0.816647181 podStartE2EDuration="816.647181ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:35:57.814949848 +0000 UTC m=+2276.805397899" watchObservedRunningTime="2026-07-08 03:35:57.816647181 +0000 UTC m=+2276.807095218" Jul 08 03:35:58.398815 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:58.398779 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:58.403695 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:58.403674 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:58.792930 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:58.792897 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:35:58.793806 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:35:58.793785 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:36:12.867894 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:12.867863 2576 generic.go:356] "Generic (PLEG): container finished" podID="95d84118-0aa0-4256-9fdb-8c996dd02365" containerID="72487042debd2eadd4c6f3693871f6646814f9501d09215098eb19dab3d2eb89" exitCode=0 Jul 08 03:36:12.868306 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:12.867933 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" event={"ID":"95d84118-0aa0-4256-9fdb-8c996dd02365","Type":"ContainerDied","Data":"72487042debd2eadd4c6f3693871f6646814f9501d09215098eb19dab3d2eb89"} Jul 08 03:36:12.868347 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:12.868314 2576 scope.go:117] "RemoveContainer" containerID="72487042debd2eadd4c6f3693871f6646814f9501d09215098eb19dab3d2eb89" Jul 08 03:36:12.869440 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:12.869414 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:36:12.869508 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:12.869482 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:36:12.869544 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:12.869514 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:36:13.793446 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:13.793409 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:36:13.878369 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:13.878333 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" event={"ID":"95d84118-0aa0-4256-9fdb-8c996dd02365","Type":"ContainerStarted","Data":"8b46999ac0d48faaf100e91a3054114d1083b070a81488b0c50bc013205a39fc"} Jul 08 03:36:14.398065 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:14.398019 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:36:14.399227 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:14.399200 2576 patch_prober.go:28] interesting pod/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 08 03:36:14.399345 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:14.399257 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" podUID="95d84118-0aa0-4256-9fdb-8c996dd02365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 08 03:36:15.399085 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:15.399023 2576 patch_prober.go:28] interesting pod/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 08 03:36:15.399549 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:15.399109 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" podUID="95d84118-0aa0-4256-9fdb-8c996dd02365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 08 03:36:16.398743 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:16.398703 2576 patch_prober.go:28] interesting pod/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 08 03:36:16.398950 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:16.398762 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" podUID="95d84118-0aa0-4256-9fdb-8c996dd02365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 08 03:36:17.398707 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:17.398664 2576 patch_prober.go:28] interesting pod/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 08 03:36:17.399121 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:17.398731 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" podUID="95d84118-0aa0-4256-9fdb-8c996dd02365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 08 03:36:18.401866 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:18.401837 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:36:18.402237 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:18.402147 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:36:18.402887 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:18.402873 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:36:20.514301 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:20.514260 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:36:41.024417 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:36:41.024386 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:37:04.131130 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:04.131033 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:37:33.116270 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:33.116225 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:37:35.275607 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:35.275565 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2"] Jul 08 03:37:35.312629 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:35.312583 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" podUID="95d84118-0aa0-4256-9fdb-8c996dd02365" containerName="istio-proxy" containerID="cri-o://8b46999ac0d48faaf100e91a3054114d1083b070a81488b0c50bc013205a39fc" gracePeriod=30 Jul 08 03:37:40.579086 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.578996 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:37:40.687905 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.687861 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-workload-certs\") pod \"95d84118-0aa0-4256-9fdb-8c996dd02365\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " Jul 08 03:37:40.687905 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.687903 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-credential-socket\") pod \"95d84118-0aa0-4256-9fdb-8c996dd02365\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " Jul 08 03:37:40.688176 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.687924 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-workload-socket\") pod \"95d84118-0aa0-4256-9fdb-8c996dd02365\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " Jul 08 03:37:40.688176 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.687947 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-envoy\") pod \"95d84118-0aa0-4256-9fdb-8c996dd02365\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " Jul 08 03:37:40.688176 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.687999 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkwlg\" (UniqueName: \"kubernetes.io/projected/95d84118-0aa0-4256-9fdb-8c996dd02365-kube-api-access-nkwlg\") pod \"95d84118-0aa0-4256-9fdb-8c996dd02365\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " Jul 08 03:37:40.688176 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.688027 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-podinfo\") pod \"95d84118-0aa0-4256-9fdb-8c996dd02365\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " Jul 08 03:37:40.688176 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.688117 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-token\") pod \"95d84118-0aa0-4256-9fdb-8c996dd02365\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " Jul 08 03:37:40.688176 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.688156 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-data\") pod \"95d84118-0aa0-4256-9fdb-8c996dd02365\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " Jul 08 03:37:40.688480 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.688270 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "95d84118-0aa0-4256-9fdb-8c996dd02365" (UID: "95d84118-0aa0-4256-9fdb-8c996dd02365"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:37:40.688480 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.688294 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d84118-0aa0-4256-9fdb-8c996dd02365-istiod-ca-cert\") pod \"95d84118-0aa0-4256-9fdb-8c996dd02365\" (UID: \"95d84118-0aa0-4256-9fdb-8c996dd02365\") " Jul 08 03:37:40.688480 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.688333 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "95d84118-0aa0-4256-9fdb-8c996dd02365" (UID: "95d84118-0aa0-4256-9fdb-8c996dd02365"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:37:40.688644 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.688592 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-workload-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:37:40.688644 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.688616 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-workload-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:37:40.688741 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.688720 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d84118-0aa0-4256-9fdb-8c996dd02365-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "95d84118-0aa0-4256-9fdb-8c996dd02365" (UID: "95d84118-0aa0-4256-9fdb-8c996dd02365"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:37:40.688801 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.688771 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "95d84118-0aa0-4256-9fdb-8c996dd02365" (UID: "95d84118-0aa0-4256-9fdb-8c996dd02365"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:37:40.689023 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.688999 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-data" (OuterVolumeSpecName: "istio-data") pod "95d84118-0aa0-4256-9fdb-8c996dd02365" (UID: "95d84118-0aa0-4256-9fdb-8c996dd02365"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:37:40.690551 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.690514 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "95d84118-0aa0-4256-9fdb-8c996dd02365" (UID: "95d84118-0aa0-4256-9fdb-8c996dd02365"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:37:40.690551 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.690528 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-token" (OuterVolumeSpecName: "istio-token") pod "95d84118-0aa0-4256-9fdb-8c996dd02365" (UID: "95d84118-0aa0-4256-9fdb-8c996dd02365"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:37:40.690964 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.690941 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "95d84118-0aa0-4256-9fdb-8c996dd02365" (UID: "95d84118-0aa0-4256-9fdb-8c996dd02365"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:37:40.690964 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.690947 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d84118-0aa0-4256-9fdb-8c996dd02365-kube-api-access-nkwlg" (OuterVolumeSpecName: "kube-api-access-nkwlg") pod "95d84118-0aa0-4256-9fdb-8c996dd02365" (UID: "95d84118-0aa0-4256-9fdb-8c996dd02365"). InnerVolumeSpecName "kube-api-access-nkwlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:37:40.789093 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.789059 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-credential-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:37:40.789093 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.789093 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-envoy\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:37:40.789304 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.789106 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkwlg\" (UniqueName: \"kubernetes.io/projected/95d84118-0aa0-4256-9fdb-8c996dd02365-kube-api-access-nkwlg\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:37:40.789304 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.789118 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-podinfo\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:37:40.789304 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.789130 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-token\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:37:40.789304 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.789140 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d84118-0aa0-4256-9fdb-8c996dd02365-istio-data\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:37:40.789304 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:40.789150 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d84118-0aa0-4256-9fdb-8c996dd02365-istiod-ca-cert\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:37:41.343981 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:41.343942 2576 generic.go:356] "Generic (PLEG): container finished" podID="95d84118-0aa0-4256-9fdb-8c996dd02365" containerID="8b46999ac0d48faaf100e91a3054114d1083b070a81488b0c50bc013205a39fc" exitCode=0 Jul 08 03:37:41.344190 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:41.343997 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" Jul 08 03:37:41.344190 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:41.344028 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" event={"ID":"95d84118-0aa0-4256-9fdb-8c996dd02365","Type":"ContainerDied","Data":"8b46999ac0d48faaf100e91a3054114d1083b070a81488b0c50bc013205a39fc"} Jul 08 03:37:41.344190 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:41.344128 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2" event={"ID":"95d84118-0aa0-4256-9fdb-8c996dd02365","Type":"ContainerDied","Data":"6c6d644a8019670f89fff1fd5cc2324ce10699e1a63cece37f086a6c6b21c9f8"} Jul 08 03:37:41.344190 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:41.344151 2576 scope.go:117] "RemoveContainer" containerID="8b46999ac0d48faaf100e91a3054114d1083b070a81488b0c50bc013205a39fc" Jul 08 03:37:41.354227 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:41.354208 2576 scope.go:117] "RemoveContainer" containerID="72487042debd2eadd4c6f3693871f6646814f9501d09215098eb19dab3d2eb89" Jul 08 03:37:41.363024 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:41.363004 2576 scope.go:117] "RemoveContainer" containerID="8b46999ac0d48faaf100e91a3054114d1083b070a81488b0c50bc013205a39fc" Jul 08 03:37:41.363362 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:37:41.363336 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b46999ac0d48faaf100e91a3054114d1083b070a81488b0c50bc013205a39fc\": container with ID starting with 8b46999ac0d48faaf100e91a3054114d1083b070a81488b0c50bc013205a39fc not found: ID does not exist" containerID="8b46999ac0d48faaf100e91a3054114d1083b070a81488b0c50bc013205a39fc" Jul 08 03:37:41.363421 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:41.363365 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b46999ac0d48faaf100e91a3054114d1083b070a81488b0c50bc013205a39fc"} err="failed to get container status \"8b46999ac0d48faaf100e91a3054114d1083b070a81488b0c50bc013205a39fc\": rpc error: code = NotFound desc = could not find container \"8b46999ac0d48faaf100e91a3054114d1083b070a81488b0c50bc013205a39fc\": container with ID starting with 8b46999ac0d48faaf100e91a3054114d1083b070a81488b0c50bc013205a39fc not found: ID does not exist" Jul 08 03:37:41.363421 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:41.363384 2576 scope.go:117] "RemoveContainer" containerID="72487042debd2eadd4c6f3693871f6646814f9501d09215098eb19dab3d2eb89" Jul 08 03:37:41.363690 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:37:41.363665 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72487042debd2eadd4c6f3693871f6646814f9501d09215098eb19dab3d2eb89\": container with ID starting with 72487042debd2eadd4c6f3693871f6646814f9501d09215098eb19dab3d2eb89 not found: ID does not exist" containerID="72487042debd2eadd4c6f3693871f6646814f9501d09215098eb19dab3d2eb89" Jul 08 03:37:41.363785 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:41.363695 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72487042debd2eadd4c6f3693871f6646814f9501d09215098eb19dab3d2eb89"} err="failed to get container status \"72487042debd2eadd4c6f3693871f6646814f9501d09215098eb19dab3d2eb89\": rpc error: code = NotFound desc = could not find container \"72487042debd2eadd4c6f3693871f6646814f9501d09215098eb19dab3d2eb89\": container with ID starting with 72487042debd2eadd4c6f3693871f6646814f9501d09215098eb19dab3d2eb89 not found: ID does not exist" Jul 08 03:37:41.367280 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:41.367260 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2"] Jul 08 03:37:41.372679 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:41.372658 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-reuse-25332d-openshift-default-9d9b85b77-l2bp2"] Jul 08 03:37:41.526816 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:41.526783 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95d84118-0aa0-4256-9fdb-8c996dd02365" path="/var/lib/kubelet/pods/95d84118-0aa0-4256-9fdb-8c996dd02365/volumes" Jul 08 03:37:51.427251 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:37:51.427212 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:38:01.651270 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:01.651239 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:38:01.656677 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:01.656651 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:38:01.657384 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:01.657358 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:38:01.662407 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:01.662387 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:38:24.713366 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:24.713286 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:38:27.410761 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:27.410728 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7"] Jul 08 03:38:27.411246 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:27.410964 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" podUID="984c5953-7506-4409-b2dd-4a0ffa7293fe" containerName="istio-proxy" containerID="cri-o://ae5e18a7128be5a8d403eb67d1a1e96ae1df970c557abf1f06fe783a5c64fc9a" gracePeriod=30 Jul 08 03:38:28.252011 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.251979 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52"] Jul 08 03:38:28.252401 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.252387 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d84118-0aa0-4256-9fdb-8c996dd02365" containerName="istio-proxy" Jul 08 03:38:28.252467 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.252403 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d84118-0aa0-4256-9fdb-8c996dd02365" containerName="istio-proxy" Jul 08 03:38:28.252467 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.252412 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d84118-0aa0-4256-9fdb-8c996dd02365" containerName="istio-proxy" Jul 08 03:38:28.252467 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.252417 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d84118-0aa0-4256-9fdb-8c996dd02365" containerName="istio-proxy" Jul 08 03:38:28.252623 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.252494 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="95d84118-0aa0-4256-9fdb-8c996dd02365" containerName="istio-proxy" Jul 08 03:38:28.254608 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.254592 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.258892 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.258871 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e8b238-openshift-default-dockercfg-x2qqp\"" Jul 08 03:38:28.269448 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.269424 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52"] Jul 08 03:38:28.309504 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.309472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-podinfo\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.309657 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.309552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-envoy\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.309657 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.309588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-credential-socket\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.309657 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.309610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istiod-ca-cert\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.309762 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.309658 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-workload-socket\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.309762 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.309694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-workload-certs\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.309762 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.309709 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-token\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.309762 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.309741 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-data\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.309886 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.309764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cgx6\" (UniqueName: \"kubernetes.io/projected/e6df0d2a-ad85-4d76-8a57-0424f09b2909-kube-api-access-9cgx6\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.410969 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.410938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-envoy\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.411459 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.410983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-credential-socket\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.411459 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.411009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istiod-ca-cert\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.411459 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.411037 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-workload-socket\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.411459 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.411103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-workload-certs\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.411459 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.411126 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-token\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.411459 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.411151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-data\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.411459 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.411182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cgx6\" (UniqueName: \"kubernetes.io/projected/e6df0d2a-ad85-4d76-8a57-0424f09b2909-kube-api-access-9cgx6\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.411459 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.411251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-podinfo\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.411886 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.411496 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-credential-socket\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.411886 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.411670 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-data\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.411886 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.411826 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-workload-certs\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.411886 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.411844 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-workload-socket\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.412204 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.412187 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istiod-ca-cert\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.413538 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.413516 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-envoy\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.414148 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.414130 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-podinfo\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.420114 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.420090 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-token\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.420364 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.420343 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cgx6\" (UniqueName: \"kubernetes.io/projected/e6df0d2a-ad85-4d76-8a57-0424f09b2909-kube-api-access-9cgx6\") pod \"e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.566729 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.566635 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:28.718216 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.718187 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52"] Jul 08 03:38:28.720457 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:38:28.720430 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6df0d2a_ad85_4d76_8a57_0424f09b2909.slice/crio-20b7ccc12b7a963760bbce0c12e6f88ecaa5482a7cc987a0d9c4f40e2bd83a16 WatchSource:0}: Error finding container 20b7ccc12b7a963760bbce0c12e6f88ecaa5482a7cc987a0d9c4f40e2bd83a16: Status 404 returned error can't find the container with id 20b7ccc12b7a963760bbce0c12e6f88ecaa5482a7cc987a0d9c4f40e2bd83a16 Jul 08 03:38:28.723078 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.723016 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:38:28.723148 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.723112 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:38:28.723148 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:28.723142 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:38:29.585286 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:29.585244 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" event={"ID":"e6df0d2a-ad85-4d76-8a57-0424f09b2909","Type":"ContainerStarted","Data":"32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058"} Jul 08 03:38:29.585695 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:29.585305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" event={"ID":"e6df0d2a-ad85-4d76-8a57-0424f09b2909","Type":"ContainerStarted","Data":"20b7ccc12b7a963760bbce0c12e6f88ecaa5482a7cc987a0d9c4f40e2bd83a16"} Jul 08 03:38:29.608704 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:29.608661 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" podStartSLOduration=1.608645583 podStartE2EDuration="1.608645583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:38:29.607450387 +0000 UTC m=+2428.597898425" watchObservedRunningTime="2026-07-08 03:38:29.608645583 +0000 UTC m=+2428.599093622" Jul 08 03:38:30.566940 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:30.566908 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:30.571691 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:30.571661 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:30.589838 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:30.589814 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:30.590537 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:30.590521 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:32.602160 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.602130 2576 generic.go:356] "Generic (PLEG): container finished" podID="984c5953-7506-4409-b2dd-4a0ffa7293fe" containerID="ae5e18a7128be5a8d403eb67d1a1e96ae1df970c557abf1f06fe783a5c64fc9a" exitCode=0 Jul 08 03:38:32.602583 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.602180 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" event={"ID":"984c5953-7506-4409-b2dd-4a0ffa7293fe","Type":"ContainerDied","Data":"ae5e18a7128be5a8d403eb67d1a1e96ae1df970c557abf1f06fe783a5c64fc9a"} Jul 08 03:38:32.602583 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.602265 2576 scope.go:117] "RemoveContainer" containerID="6e621d53b3395a5adc9ee906e8bc4966bfcac1c963d57fe8d4dac7d2fd8a8816" Jul 08 03:38:32.669981 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.669957 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:38:32.752317 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.752217 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-credential-socket\") pod \"984c5953-7506-4409-b2dd-4a0ffa7293fe\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " Jul 08 03:38:32.752317 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.752261 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-podinfo\") pod \"984c5953-7506-4409-b2dd-4a0ffa7293fe\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " Jul 08 03:38:32.752568 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.752333 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-workload-socket\") pod \"984c5953-7506-4409-b2dd-4a0ffa7293fe\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " Jul 08 03:38:32.752568 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.752382 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-data\") pod \"984c5953-7506-4409-b2dd-4a0ffa7293fe\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " Jul 08 03:38:32.752568 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.752510 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-workload-certs\") pod \"984c5953-7506-4409-b2dd-4a0ffa7293fe\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " Jul 08 03:38:32.752568 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.752550 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zjfb\" (UniqueName: \"kubernetes.io/projected/984c5953-7506-4409-b2dd-4a0ffa7293fe-kube-api-access-4zjfb\") pod \"984c5953-7506-4409-b2dd-4a0ffa7293fe\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " Jul 08 03:38:32.752777 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.752583 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-envoy\") pod \"984c5953-7506-4409-b2dd-4a0ffa7293fe\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " Jul 08 03:38:32.752777 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.752616 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-token\") pod \"984c5953-7506-4409-b2dd-4a0ffa7293fe\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " Jul 08 03:38:32.752777 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.752646 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984c5953-7506-4409-b2dd-4a0ffa7293fe-istiod-ca-cert\") pod \"984c5953-7506-4409-b2dd-4a0ffa7293fe\" (UID: \"984c5953-7506-4409-b2dd-4a0ffa7293fe\") " Jul 08 03:38:32.752777 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.752611 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "984c5953-7506-4409-b2dd-4a0ffa7293fe" (UID: "984c5953-7506-4409-b2dd-4a0ffa7293fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:38:32.752991 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.752814 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "984c5953-7506-4409-b2dd-4a0ffa7293fe" (UID: "984c5953-7506-4409-b2dd-4a0ffa7293fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:38:32.753132 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.753097 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-credential-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:38:32.753132 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.753123 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-workload-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:38:32.753328 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.753157 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984c5953-7506-4409-b2dd-4a0ffa7293fe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "984c5953-7506-4409-b2dd-4a0ffa7293fe" (UID: "984c5953-7506-4409-b2dd-4a0ffa7293fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:38:32.753328 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.753232 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-data" (OuterVolumeSpecName: "istio-data") pod "984c5953-7506-4409-b2dd-4a0ffa7293fe" (UID: "984c5953-7506-4409-b2dd-4a0ffa7293fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:38:32.753608 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.753351 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "984c5953-7506-4409-b2dd-4a0ffa7293fe" (UID: "984c5953-7506-4409-b2dd-4a0ffa7293fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:38:32.755091 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.755024 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984c5953-7506-4409-b2dd-4a0ffa7293fe-kube-api-access-4zjfb" (OuterVolumeSpecName: "kube-api-access-4zjfb") pod "984c5953-7506-4409-b2dd-4a0ffa7293fe" (UID: "984c5953-7506-4409-b2dd-4a0ffa7293fe"). InnerVolumeSpecName "kube-api-access-4zjfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:38:32.755205 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.755150 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "984c5953-7506-4409-b2dd-4a0ffa7293fe" (UID: "984c5953-7506-4409-b2dd-4a0ffa7293fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:38:32.755721 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.755697 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "984c5953-7506-4409-b2dd-4a0ffa7293fe" (UID: "984c5953-7506-4409-b2dd-4a0ffa7293fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:38:32.755820 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.755713 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-token" (OuterVolumeSpecName: "istio-token") pod "984c5953-7506-4409-b2dd-4a0ffa7293fe" (UID: "984c5953-7506-4409-b2dd-4a0ffa7293fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:38:32.854393 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.854362 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984c5953-7506-4409-b2dd-4a0ffa7293fe-istiod-ca-cert\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:38:32.854393 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.854389 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-podinfo\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:38:32.854393 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.854398 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-data\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:38:32.854631 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.854405 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-workload-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:38:32.854631 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.854413 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zjfb\" (UniqueName: \"kubernetes.io/projected/984c5953-7506-4409-b2dd-4a0ffa7293fe-kube-api-access-4zjfb\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:38:32.854631 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.854421 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-envoy\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:38:32.854631 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:32.854437 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984c5953-7506-4409-b2dd-4a0ffa7293fe-istio-token\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:38:33.608995 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:33.608966 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" Jul 08 03:38:33.609430 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:33.608972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7" event={"ID":"984c5953-7506-4409-b2dd-4a0ffa7293fe","Type":"ContainerDied","Data":"ad4e3f4fd219a3ad9d907208b29438862b7a2a7e8ec2e98e287414966c09850c"} Jul 08 03:38:33.609430 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:33.609112 2576 scope.go:117] "RemoveContainer" containerID="ae5e18a7128be5a8d403eb67d1a1e96ae1df970c557abf1f06fe783a5c64fc9a" Jul 08 03:38:33.629668 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:33.629629 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7"] Jul 08 03:38:33.632018 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:33.631996 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7142f2-openshift-default-7c868f9699-dvmr7"] Jul 08 03:38:35.525464 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:35.525431 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="984c5953-7506-4409-b2dd-4a0ffa7293fe" path="/var/lib/kubelet/pods/984c5953-7506-4409-b2dd-4a0ffa7293fe/volumes" Jul 08 03:38:36.785440 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.785405 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch"] Jul 08 03:38:36.785835 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.785821 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="984c5953-7506-4409-b2dd-4a0ffa7293fe" containerName="istio-proxy" Jul 08 03:38:36.785879 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.785837 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="984c5953-7506-4409-b2dd-4a0ffa7293fe" containerName="istio-proxy" Jul 08 03:38:36.785879 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.785853 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="984c5953-7506-4409-b2dd-4a0ffa7293fe" containerName="istio-proxy" Jul 08 03:38:36.785879 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.785861 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="984c5953-7506-4409-b2dd-4a0ffa7293fe" containerName="istio-proxy" Jul 08 03:38:36.785971 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.785942 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="95d84118-0aa0-4256-9fdb-8c996dd02365" containerName="istio-proxy" Jul 08 03:38:36.785971 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.785959 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="984c5953-7506-4409-b2dd-4a0ffa7293fe" containerName="istio-proxy" Jul 08 03:38:36.785971 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.785967 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="984c5953-7506-4409-b2dd-4a0ffa7293fe" containerName="istio-proxy" Jul 08 03:38:36.789385 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.789364 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.793022 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.793001 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-da32fa-openshift-default-dockercfg-vb9px\"" Jul 08 03:38:36.804941 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.804918 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch"] Jul 08 03:38:36.891829 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.891794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-workload-certs\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.892057 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.891848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jqvf\" (UniqueName: \"kubernetes.io/projected/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-kube-api-access-5jqvf\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.892057 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.891878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-token\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.892057 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.891935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-podinfo\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.892057 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.891977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-workload-socket\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.892057 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.892001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-envoy\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.892057 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.892023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-data\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.892364 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.892077 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istiod-ca-cert\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.892364 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.892160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-credential-socket\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.993169 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.993131 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-workload-certs\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.993169 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.993172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jqvf\" (UniqueName: \"kubernetes.io/projected/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-kube-api-access-5jqvf\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.993477 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.993202 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-token\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.993477 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.993255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-podinfo\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.993477 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.993286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-workload-socket\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.993477 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.993310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-envoy\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.993477 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.993442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-data\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.993477 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.993490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istiod-ca-cert\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.993835 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.993544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-credential-socket\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.993835 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.993665 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-workload-socket\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.994031 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.993829 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-data\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.994031 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.993888 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-credential-socket\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.994031 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.993941 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-workload-certs\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.994620 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.994594 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istiod-ca-cert\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.995696 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.995646 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-envoy\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:36.996017 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:36.995998 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-podinfo\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:37.002317 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:37.002289 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-token\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:37.002663 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:37.002640 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jqvf\" (UniqueName: \"kubernetes.io/projected/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-kube-api-access-5jqvf\") pod \"e2e-api-b-da32fa-openshift-default-7c89774975-txnch\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:37.100743 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:37.100649 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:37.252337 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:37.252312 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch"] Jul 08 03:38:37.254788 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:38:37.254760 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21b287b_e9a6_4aa2_bdbe_d08b24308fbf.slice/crio-17c4e8f984f46da69fb0719ae9850f91bdeab80e852dc839afd6be0e762895bf WatchSource:0}: Error finding container 17c4e8f984f46da69fb0719ae9850f91bdeab80e852dc839afd6be0e762895bf: Status 404 returned error can't find the container with id 17c4e8f984f46da69fb0719ae9850f91bdeab80e852dc839afd6be0e762895bf Jul 08 03:38:37.257706 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:37.257664 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:38:37.257795 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:37.257760 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:38:37.257842 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:37.257811 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:38:37.631117 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:37.631075 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" event={"ID":"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf","Type":"ContainerStarted","Data":"b0161bed3816f965821df07a5795fdd012ceda7f6233c8b0cc182551571c7130"} Jul 08 03:38:37.631315 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:37.631139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" event={"ID":"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf","Type":"ContainerStarted","Data":"17c4e8f984f46da69fb0719ae9850f91bdeab80e852dc839afd6be0e762895bf"} Jul 08 03:38:37.655343 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:37.655286 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" podStartSLOduration=1.6552671129999998 podStartE2EDuration="1.655267113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:38:37.652701008 +0000 UTC m=+2436.643149046" watchObservedRunningTime="2026-07-08 03:38:37.655267113 +0000 UTC m=+2436.645715151" Jul 08 03:38:38.101574 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:38.101541 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:38.106557 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:38.106534 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:38.635565 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:38.635538 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:38.636605 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:38.636587 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:39.425616 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:39.425582 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:38:46.631252 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:46.631207 2576 patch_prober.go:28] interesting pod/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 03:38:46.631804 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:46.631297 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" podUID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 03:38:47.494260 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:38:47.494221 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6df0d2a_ad85_4d76_8a57_0424f09b2909.slice/crio-conmon-32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058.scope\": RecentStats: unable to find data in memory cache]" Jul 08 03:38:47.494260 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:38:47.494247 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6df0d2a_ad85_4d76_8a57_0424f09b2909.slice/crio-conmon-32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058.scope\": RecentStats: unable to find data in memory cache]" Jul 08 03:38:47.688692 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:47.688656 2576 generic.go:356] "Generic (PLEG): container finished" podID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" containerID="32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058" exitCode=0 Jul 08 03:38:47.689091 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:47.688729 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" event={"ID":"e6df0d2a-ad85-4d76-8a57-0424f09b2909","Type":"ContainerDied","Data":"32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058"} Jul 08 03:38:47.689160 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:47.689147 2576 scope.go:117] "RemoveContainer" containerID="32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058" Jul 08 03:38:47.690228 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:47.690192 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:38:47.690333 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:47.690281 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:38:47.690333 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:47.690315 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:38:48.696386 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:48.696351 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" event={"ID":"e6df0d2a-ad85-4d76-8a57-0424f09b2909","Type":"ContainerStarted","Data":"11c536dea9e3b50499476bad7a854fc9d559a808c642195aa83829e165fa962a"} Jul 08 03:38:49.571068 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:49.570401 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:49.571068 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:49.570529 2576 patch_prober.go:28] interesting pod/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 08 03:38:49.571068 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:49.570570 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" podUID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 08 03:38:50.567727 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:50.567686 2576 patch_prober.go:28] interesting pod/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 08 03:38:50.568365 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:50.568333 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" podUID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 08 03:38:51.567854 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:51.567817 2576 patch_prober.go:28] interesting pod/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 08 03:38:51.568517 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:51.568482 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" podUID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 08 03:38:52.567725 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:52.567686 2576 patch_prober.go:28] interesting pod/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 08 03:38:52.568037 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:52.568006 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" podUID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 08 03:38:53.576620 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:53.576586 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:53.577897 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:53.577872 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:53.579770 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:53.579743 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:38:56.742865 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:56.742831 2576 generic.go:356] "Generic (PLEG): container finished" podID="a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" containerID="b0161bed3816f965821df07a5795fdd012ceda7f6233c8b0cc182551571c7130" exitCode=0 Jul 08 03:38:56.743292 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:56.742905 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" event={"ID":"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf","Type":"ContainerDied","Data":"b0161bed3816f965821df07a5795fdd012ceda7f6233c8b0cc182551571c7130"} Jul 08 03:38:56.743364 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:56.743345 2576 scope.go:117] "RemoveContainer" containerID="b0161bed3816f965821df07a5795fdd012ceda7f6233c8b0cc182551571c7130" Jul 08 03:38:56.744399 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:56.744369 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:38:56.744491 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:56.744456 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:38:56.744529 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:56.744503 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:38:57.314983 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:57.314948 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:38:57.749666 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:57.749633 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" event={"ID":"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf","Type":"ContainerStarted","Data":"66fdb9b9012306c164244facc69b6b32a682405fd624be42db66074bbe5f4ee6"} Jul 08 03:38:58.101429 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:58.101335 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:38:58.101576 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:58.101529 2576 patch_prober.go:28] interesting pod/e2e-api-b-da32fa-openshift-default-7c89774975-txnch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 08 03:38:58.101621 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:58.101580 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" podUID="a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 08 03:38:59.101425 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:59.101387 2576 patch_prober.go:28] interesting pod/e2e-api-b-da32fa-openshift-default-7c89774975-txnch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 08 03:38:59.101904 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:59.101460 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" podUID="a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 08 03:38:59.102675 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:59.102649 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52"] Jul 08 03:38:59.102872 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:38:59.102827 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" podUID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" containerName="istio-proxy" containerID="cri-o://11c536dea9e3b50499476bad7a854fc9d559a808c642195aa83829e165fa962a" gracePeriod=30 Jul 08 03:39:00.101162 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:00.101125 2576 patch_prober.go:28] interesting pod/e2e-api-b-da32fa-openshift-default-7c89774975-txnch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 08 03:39:00.101343 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:00.101187 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" podUID="a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 08 03:39:01.101359 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:01.101323 2576 patch_prober.go:28] interesting pod/e2e-api-b-da32fa-openshift-default-7c89774975-txnch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 08 03:39:01.101722 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:01.101376 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" podUID="a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 08 03:39:02.104596 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:02.104516 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:39:02.104992 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:02.104787 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:39:02.105543 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:02.105524 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:39:04.369109 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.369025 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:39:04.441891 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.441864 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-envoy\") pod \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " Jul 08 03:39:04.442086 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.441904 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-workload-socket\") pod \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " Jul 08 03:39:04.442086 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.441934 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-workload-certs\") pod \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " Jul 08 03:39:04.442086 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.441980 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-credential-socket\") pod \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " Jul 08 03:39:04.442086 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.441995 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istiod-ca-cert\") pod \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " Jul 08 03:39:04.442086 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.442012 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-token\") pod \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " Jul 08 03:39:04.442086 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.442074 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-data\") pod \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " Jul 08 03:39:04.442442 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.442096 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cgx6\" (UniqueName: \"kubernetes.io/projected/e6df0d2a-ad85-4d76-8a57-0424f09b2909-kube-api-access-9cgx6\") pod \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " Jul 08 03:39:04.442442 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.442129 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-podinfo\") pod \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\" (UID: \"e6df0d2a-ad85-4d76-8a57-0424f09b2909\") " Jul 08 03:39:04.442442 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.442200 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e6df0d2a-ad85-4d76-8a57-0424f09b2909" (UID: "e6df0d2a-ad85-4d76-8a57-0424f09b2909"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:39:04.442442 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.442313 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e6df0d2a-ad85-4d76-8a57-0424f09b2909" (UID: "e6df0d2a-ad85-4d76-8a57-0424f09b2909"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:39:04.442442 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.442365 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e6df0d2a-ad85-4d76-8a57-0424f09b2909" (UID: "e6df0d2a-ad85-4d76-8a57-0424f09b2909"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:39:04.442903 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.442469 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-workload-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:04.442903 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.442487 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-credential-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:04.442903 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.442496 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istiod-ca-cert\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:04.442903 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.442653 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e6df0d2a-ad85-4d76-8a57-0424f09b2909" (UID: "e6df0d2a-ad85-4d76-8a57-0424f09b2909"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:39:04.443145 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.442985 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-data" (OuterVolumeSpecName: "istio-data") pod "e6df0d2a-ad85-4d76-8a57-0424f09b2909" (UID: "e6df0d2a-ad85-4d76-8a57-0424f09b2909"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:39:04.444409 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.444383 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e6df0d2a-ad85-4d76-8a57-0424f09b2909" (UID: "e6df0d2a-ad85-4d76-8a57-0424f09b2909"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:39:04.444513 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.444480 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-token" (OuterVolumeSpecName: "istio-token") pod "e6df0d2a-ad85-4d76-8a57-0424f09b2909" (UID: "e6df0d2a-ad85-4d76-8a57-0424f09b2909"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:39:04.444891 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.444866 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e6df0d2a-ad85-4d76-8a57-0424f09b2909" (UID: "e6df0d2a-ad85-4d76-8a57-0424f09b2909"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:39:04.444891 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.444871 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6df0d2a-ad85-4d76-8a57-0424f09b2909-kube-api-access-9cgx6" (OuterVolumeSpecName: "kube-api-access-9cgx6") pod "e6df0d2a-ad85-4d76-8a57-0424f09b2909" (UID: "e6df0d2a-ad85-4d76-8a57-0424f09b2909"). InnerVolumeSpecName "kube-api-access-9cgx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:39:04.542899 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.542869 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-envoy\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:04.542899 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.542892 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-workload-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:04.542899 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.542901 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-token\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:04.542899 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.542908 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-data\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:04.543179 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.542916 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cgx6\" (UniqueName: \"kubernetes.io/projected/e6df0d2a-ad85-4d76-8a57-0424f09b2909-kube-api-access-9cgx6\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:04.543179 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.542924 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6df0d2a-ad85-4d76-8a57-0424f09b2909-istio-podinfo\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:04.792354 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.792325 2576 generic.go:356] "Generic (PLEG): container finished" podID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" containerID="11c536dea9e3b50499476bad7a854fc9d559a808c642195aa83829e165fa962a" exitCode=0 Jul 08 03:39:04.792535 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.792378 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" Jul 08 03:39:04.792535 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.792422 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" event={"ID":"e6df0d2a-ad85-4d76-8a57-0424f09b2909","Type":"ContainerDied","Data":"11c536dea9e3b50499476bad7a854fc9d559a808c642195aa83829e165fa962a"} Jul 08 03:39:04.792535 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.792477 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52" event={"ID":"e6df0d2a-ad85-4d76-8a57-0424f09b2909","Type":"ContainerDied","Data":"20b7ccc12b7a963760bbce0c12e6f88ecaa5482a7cc987a0d9c4f40e2bd83a16"} Jul 08 03:39:04.792535 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.792506 2576 scope.go:117] "RemoveContainer" containerID="11c536dea9e3b50499476bad7a854fc9d559a808c642195aa83829e165fa962a" Jul 08 03:39:04.808871 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.808852 2576 scope.go:117] "RemoveContainer" containerID="32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058" Jul 08 03:39:04.817264 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.817244 2576 scope.go:117] "RemoveContainer" containerID="11c536dea9e3b50499476bad7a854fc9d559a808c642195aa83829e165fa962a" Jul 08 03:39:04.817515 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:39:04.817498 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c536dea9e3b50499476bad7a854fc9d559a808c642195aa83829e165fa962a\": container with ID starting with 11c536dea9e3b50499476bad7a854fc9d559a808c642195aa83829e165fa962a not found: ID does not exist" containerID="11c536dea9e3b50499476bad7a854fc9d559a808c642195aa83829e165fa962a" Jul 08 03:39:04.817565 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.817522 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c536dea9e3b50499476bad7a854fc9d559a808c642195aa83829e165fa962a"} err="failed to get container status \"11c536dea9e3b50499476bad7a854fc9d559a808c642195aa83829e165fa962a\": rpc error: code = NotFound desc = could not find container \"11c536dea9e3b50499476bad7a854fc9d559a808c642195aa83829e165fa962a\": container with ID starting with 11c536dea9e3b50499476bad7a854fc9d559a808c642195aa83829e165fa962a not found: ID does not exist" Jul 08 03:39:04.817565 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.817538 2576 scope.go:117] "RemoveContainer" containerID="32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058" Jul 08 03:39:04.817744 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:39:04.817729 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058\": container with ID starting with 32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058 not found: ID does not exist" containerID="32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058" Jul 08 03:39:04.817790 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.817748 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058"} err="failed to get container status \"32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058\": rpc error: code = NotFound desc = could not find container \"32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058\": container with ID starting with 32d98a96b5a603140c8f12de31b1cd462277563311ca4f529c17067ab8e64058 not found: ID does not exist" Jul 08 03:39:04.822396 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.822372 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52"] Jul 08 03:39:04.826129 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:04.826104 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e8b238-openshift-default-67464b8b7f-nwt52"] Jul 08 03:39:05.526832 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:05.526797 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" path="/var/lib/kubelet/pods/e6df0d2a-ad85-4d76-8a57-0424f09b2909/volumes" Jul 08 03:39:06.173826 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:06.173794 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch"] Jul 08 03:39:06.174033 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:06.173992 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" podUID="a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" containerName="istio-proxy" containerID="cri-o://66fdb9b9012306c164244facc69b6b32a682405fd624be42db66074bbe5f4ee6" gracePeriod=30 Jul 08 03:39:07.056868 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.056837 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq"] Jul 08 03:39:07.057339 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.057250 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" containerName="istio-proxy" Jul 08 03:39:07.057339 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.057264 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" containerName="istio-proxy" Jul 08 03:39:07.057339 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.057280 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" containerName="istio-proxy" Jul 08 03:39:07.057339 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.057285 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" containerName="istio-proxy" Jul 08 03:39:07.057553 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.057355 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" containerName="istio-proxy" Jul 08 03:39:07.057553 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.057367 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6df0d2a-ad85-4d76-8a57-0424f09b2909" containerName="istio-proxy" Jul 08 03:39:07.060612 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.060590 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.063524 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.063506 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-788e6a-openshift-default-dockercfg-67fzv\"" Jul 08 03:39:07.075833 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.075811 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq"] Jul 08 03:39:07.168664 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.168630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-data\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.168664 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.168669 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istiod-ca-cert\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.168920 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.168693 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-podinfo\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.168920 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.168808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-workload-socket\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.168920 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.168848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-credential-socket\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.169037 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.168945 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-workload-certs\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.169037 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.169011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqbgp\" (UniqueName: \"kubernetes.io/projected/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-kube-api-access-zqbgp\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.169149 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.169091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-envoy\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.169149 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.169120 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-token\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.270014 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.269982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-token\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.270185 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.270026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-data\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.270185 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.270060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istiod-ca-cert\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.270185 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.270080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-podinfo\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.270185 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.270127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-workload-socket\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.270185 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.270142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-credential-socket\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.270480 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.270194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-workload-certs\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.270480 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.270249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqbgp\" (UniqueName: \"kubernetes.io/projected/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-kube-api-access-zqbgp\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.270480 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.270294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-envoy\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.270480 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.270472 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-data\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.270696 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.270619 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-workload-socket\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.270867 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.270839 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-credential-socket\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.270997 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.270898 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-workload-certs\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.271138 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.271117 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istiod-ca-cert\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.272822 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.272799 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-envoy\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.273060 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.273023 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-podinfo\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.279388 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.279364 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-token\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.279513 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.279490 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqbgp\" (UniqueName: \"kubernetes.io/projected/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-kube-api-access-zqbgp\") pod \"e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.371014 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.370942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:07.531200 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.531180 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq"] Jul 08 03:39:07.533072 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:39:07.533026 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cabf3e4_1be3_4b45_bfe2_f547e3e069cf.slice/crio-de397e8b568c420d114bf4eea82f33fafadfcd19d0f32dc05c0b0332f2a39c06 WatchSource:0}: Error finding container de397e8b568c420d114bf4eea82f33fafadfcd19d0f32dc05c0b0332f2a39c06: Status 404 returned error can't find the container with id de397e8b568c420d114bf4eea82f33fafadfcd19d0f32dc05c0b0332f2a39c06 Jul 08 03:39:07.535330 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.535297 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:39:07.535410 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.535360 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:39:07.535410 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.535387 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:39:07.811178 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.811138 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" event={"ID":"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf","Type":"ContainerStarted","Data":"cae2be4f4fa3e008b16cb94e3a249cb2273d9b4093a79303e6edc4fec0cadb2b"} Jul 08 03:39:07.811359 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.811203 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" event={"ID":"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf","Type":"ContainerStarted","Data":"de397e8b568c420d114bf4eea82f33fafadfcd19d0f32dc05c0b0332f2a39c06"} Jul 08 03:39:07.833217 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:07.833149 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" podStartSLOduration=0.833127714 podStartE2EDuration="833.127714ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:39:07.831013798 +0000 UTC m=+2466.821461836" watchObservedRunningTime="2026-07-08 03:39:07.833127714 +0000 UTC m=+2466.823575755" Jul 08 03:39:08.372115 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:08.372084 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:08.377381 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:08.377357 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:08.816405 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:08.816376 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:08.817281 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:08.817261 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:10.822613 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:10.822575 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:39:11.436901 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.436847 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:39:11.511434 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.511396 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-envoy\") pod \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " Jul 08 03:39:11.511434 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.511438 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-credential-socket\") pod \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " Jul 08 03:39:11.511684 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.511476 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-data\") pod \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " Jul 08 03:39:11.511684 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.511497 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-workload-certs\") pod \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " Jul 08 03:39:11.511684 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.511514 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jqvf\" (UniqueName: \"kubernetes.io/projected/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-kube-api-access-5jqvf\") pod \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " Jul 08 03:39:11.511684 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.511547 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-podinfo\") pod \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " Jul 08 03:39:11.511684 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.511575 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-token\") pod \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " Jul 08 03:39:11.511684 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.511599 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-workload-socket\") pod \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " Jul 08 03:39:11.511684 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.511624 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istiod-ca-cert\") pod \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\" (UID: \"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf\") " Jul 08 03:39:11.512031 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.511937 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" (UID: "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:39:11.512217 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.512180 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" (UID: "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:39:11.512321 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.512216 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" (UID: "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:39:11.512321 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.512300 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" (UID: "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:39:11.512462 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.512437 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-data" (OuterVolumeSpecName: "istio-data") pod "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" (UID: "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:39:11.514187 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.514160 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-token" (OuterVolumeSpecName: "istio-token") pod "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" (UID: "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:39:11.514284 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.514270 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" (UID: "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:39:11.514464 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.514444 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" (UID: "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:39:11.514527 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.514467 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-kube-api-access-5jqvf" (OuterVolumeSpecName: "kube-api-access-5jqvf") pod "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" (UID: "a21b287b-e9a6-4aa2-bdbe-d08b24308fbf"). InnerVolumeSpecName "kube-api-access-5jqvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:39:11.612890 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.612854 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-podinfo\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:11.612890 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.612891 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-token\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:11.613150 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.612906 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-workload-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:11.613150 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.612919 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istiod-ca-cert\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:11.613150 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.612931 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-envoy\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:11.613150 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.612943 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-credential-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:11.613150 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.612956 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-istio-data\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:11.613150 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.612971 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-workload-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:11.613150 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.612983 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jqvf\" (UniqueName: \"kubernetes.io/projected/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf-kube-api-access-5jqvf\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:39:11.832929 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.832828 2576 generic.go:356] "Generic (PLEG): container finished" podID="a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" containerID="66fdb9b9012306c164244facc69b6b32a682405fd624be42db66074bbe5f4ee6" exitCode=0 Jul 08 03:39:11.832929 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.832875 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" Jul 08 03:39:11.832929 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.832917 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" event={"ID":"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf","Type":"ContainerDied","Data":"66fdb9b9012306c164244facc69b6b32a682405fd624be42db66074bbe5f4ee6"} Jul 08 03:39:11.833461 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.832978 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch" event={"ID":"a21b287b-e9a6-4aa2-bdbe-d08b24308fbf","Type":"ContainerDied","Data":"17c4e8f984f46da69fb0719ae9850f91bdeab80e852dc839afd6be0e762895bf"} Jul 08 03:39:11.833461 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.833004 2576 scope.go:117] "RemoveContainer" containerID="66fdb9b9012306c164244facc69b6b32a682405fd624be42db66074bbe5f4ee6" Jul 08 03:39:11.842152 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.842132 2576 scope.go:117] "RemoveContainer" containerID="b0161bed3816f965821df07a5795fdd012ceda7f6233c8b0cc182551571c7130" Jul 08 03:39:11.850537 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.850516 2576 scope.go:117] "RemoveContainer" containerID="66fdb9b9012306c164244facc69b6b32a682405fd624be42db66074bbe5f4ee6" Jul 08 03:39:11.850818 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:39:11.850798 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66fdb9b9012306c164244facc69b6b32a682405fd624be42db66074bbe5f4ee6\": container with ID starting with 66fdb9b9012306c164244facc69b6b32a682405fd624be42db66074bbe5f4ee6 not found: ID does not exist" containerID="66fdb9b9012306c164244facc69b6b32a682405fd624be42db66074bbe5f4ee6" Jul 08 03:39:11.850888 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.850844 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66fdb9b9012306c164244facc69b6b32a682405fd624be42db66074bbe5f4ee6"} err="failed to get container status \"66fdb9b9012306c164244facc69b6b32a682405fd624be42db66074bbe5f4ee6\": rpc error: code = NotFound desc = could not find container \"66fdb9b9012306c164244facc69b6b32a682405fd624be42db66074bbe5f4ee6\": container with ID starting with 66fdb9b9012306c164244facc69b6b32a682405fd624be42db66074bbe5f4ee6 not found: ID does not exist" Jul 08 03:39:11.850888 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.850862 2576 scope.go:117] "RemoveContainer" containerID="b0161bed3816f965821df07a5795fdd012ceda7f6233c8b0cc182551571c7130" Jul 08 03:39:11.851138 ip-10-0-142-99 kubenswrapper[2576]: E0708 03:39:11.851118 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0161bed3816f965821df07a5795fdd012ceda7f6233c8b0cc182551571c7130\": container with ID starting with b0161bed3816f965821df07a5795fdd012ceda7f6233c8b0cc182551571c7130 not found: ID does not exist" containerID="b0161bed3816f965821df07a5795fdd012ceda7f6233c8b0cc182551571c7130" Jul 08 03:39:11.851193 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.851146 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0161bed3816f965821df07a5795fdd012ceda7f6233c8b0cc182551571c7130"} err="failed to get container status \"b0161bed3816f965821df07a5795fdd012ceda7f6233c8b0cc182551571c7130\": rpc error: code = NotFound desc = could not find container \"b0161bed3816f965821df07a5795fdd012ceda7f6233c8b0cc182551571c7130\": container with ID starting with b0161bed3816f965821df07a5795fdd012ceda7f6233c8b0cc182551571c7130 not found: ID does not exist" Jul 08 03:39:11.862021 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.861998 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch"] Jul 08 03:39:11.872193 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:11.872170 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-da32fa-openshift-default-7c89774975-txnch"] Jul 08 03:39:13.525804 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:13.525775 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" path="/var/lib/kubelet/pods/a21b287b-e9a6-4aa2-bdbe-d08b24308fbf/volumes" Jul 08 03:39:22.417382 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:22.417345 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:39:22.907337 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:22.907251 2576 generic.go:356] "Generic (PLEG): container finished" podID="1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" containerID="cae2be4f4fa3e008b16cb94e3a249cb2273d9b4093a79303e6edc4fec0cadb2b" exitCode=0 Jul 08 03:39:22.907337 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:22.907297 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" event={"ID":"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf","Type":"ContainerDied","Data":"cae2be4f4fa3e008b16cb94e3a249cb2273d9b4093a79303e6edc4fec0cadb2b"} Jul 08 03:39:22.907674 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:22.907660 2576 scope.go:117] "RemoveContainer" containerID="cae2be4f4fa3e008b16cb94e3a249cb2273d9b4093a79303e6edc4fec0cadb2b" Jul 08 03:39:22.908775 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:22.908744 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:39:22.908833 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:22.908821 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:39:22.908869 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:22.908856 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:39:23.817306 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:23.817271 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:23.914636 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:23.914601 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" event={"ID":"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf","Type":"ContainerStarted","Data":"207566171fd963f2cbff58ad5fc1bb6672a69e4d9bcd5889aa334dd95c4bebfc"} Jul 08 03:39:24.373163 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:24.371989 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:24.373452 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:24.373427 2576 patch_prober.go:28] interesting pod/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 08 03:39:24.373629 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:24.373604 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" podUID="1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 08 03:39:25.372210 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:25.372170 2576 patch_prober.go:28] interesting pod/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 08 03:39:25.372794 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:25.372758 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" podUID="1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 08 03:39:26.371844 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:26.371805 2576 patch_prober.go:28] interesting pod/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 08 03:39:26.372196 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:26.372161 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" podUID="1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 08 03:39:27.371841 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:27.371791 2576 patch_prober.go:28] interesting pod/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 08 03:39:27.372410 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:27.371874 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" podUID="1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 08 03:39:28.376000 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:28.375964 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:28.376476 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:28.376285 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:39:28.377087 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:39:28.377060 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:42:31.404620 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:31.404580 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:42:33.662744 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:33.662709 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq"] Jul 08 03:42:33.663197 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:33.662963 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" podUID="1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" containerName="istio-proxy" containerID="cri-o://207566171fd963f2cbff58ad5fc1bb6672a69e4d9bcd5889aa334dd95c4bebfc" gracePeriod=30 Jul 08 03:42:38.900302 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:38.900262 2576 generic.go:356] "Generic (PLEG): container finished" podID="1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" containerID="207566171fd963f2cbff58ad5fc1bb6672a69e4d9bcd5889aa334dd95c4bebfc" exitCode=0 Jul 08 03:42:38.900819 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:38.900337 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" event={"ID":"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf","Type":"ContainerDied","Data":"207566171fd963f2cbff58ad5fc1bb6672a69e4d9bcd5889aa334dd95c4bebfc"} Jul 08 03:42:38.900819 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:38.900429 2576 scope.go:117] "RemoveContainer" containerID="cae2be4f4fa3e008b16cb94e3a249cb2273d9b4093a79303e6edc4fec0cadb2b" Jul 08 03:42:38.935121 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:38.935030 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:42:39.061321 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.061291 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-data\") pod \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " Jul 08 03:42:39.061529 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.061356 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-workload-certs\") pod \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " Jul 08 03:42:39.061529 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.061412 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqbgp\" (UniqueName: \"kubernetes.io/projected/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-kube-api-access-zqbgp\") pod \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " Jul 08 03:42:39.061529 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.061441 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-podinfo\") pod \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " Jul 08 03:42:39.061529 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.061468 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-workload-socket\") pod \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " Jul 08 03:42:39.061529 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.061503 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istiod-ca-cert\") pod \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " Jul 08 03:42:39.061529 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.061525 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-token\") pod \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " Jul 08 03:42:39.061854 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.061572 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-credential-socket\") pod \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " Jul 08 03:42:39.061854 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.061600 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-envoy\") pod \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\" (UID: \"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf\") " Jul 08 03:42:39.061854 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.061653 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" (UID: "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:42:39.062014 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.061903 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-workload-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:42:39.062087 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.062026 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" (UID: "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:42:39.062087 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.062071 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" (UID: "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:42:39.062341 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.062299 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" (UID: "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:42:39.062341 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.062307 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-data" (OuterVolumeSpecName: "istio-data") pod "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" (UID: "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:42:39.064303 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.064267 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" (UID: "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:42:39.064401 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.064337 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-kube-api-access-zqbgp" (OuterVolumeSpecName: "kube-api-access-zqbgp") pod "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" (UID: "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf"). InnerVolumeSpecName "kube-api-access-zqbgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:42:39.064482 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.064456 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" (UID: "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:42:39.064587 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.064509 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" (UID: "1cabf3e4-1be3-4b45-bfe2-f547e3e069cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:42:39.162819 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.162781 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqbgp\" (UniqueName: \"kubernetes.io/projected/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-kube-api-access-zqbgp\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:42:39.162819 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.162810 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-podinfo\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:42:39.162819 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.162819 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-workload-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:42:39.163083 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.162829 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istiod-ca-cert\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:42:39.163083 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.162837 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-token\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:42:39.163083 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.162845 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-credential-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:42:39.163083 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.162853 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-envoy\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:42:39.163083 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.162860 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf-istio-data\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\"" Jul 08 03:42:39.907686 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.907602 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" Jul 08 03:42:39.907686 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.907622 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq" event={"ID":"1cabf3e4-1be3-4b45-bfe2-f547e3e069cf","Type":"ContainerDied","Data":"de397e8b568c420d114bf4eea82f33fafadfcd19d0f32dc05c0b0332f2a39c06"} Jul 08 03:42:39.907686 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.907678 2576 scope.go:117] "RemoveContainer" containerID="207566171fd963f2cbff58ad5fc1bb6672a69e4d9bcd5889aa334dd95c4bebfc" Jul 08 03:42:39.925288 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.925255 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq"] Jul 08 03:42:39.928543 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:39.928516 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-788e6a-openshift-default-c84d478c-5glqq"] Jul 08 03:42:41.525917 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:41.525877 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" path="/var/lib/kubelet/pods/1cabf3e4-1be3-4b45-bfe2-f547e3e069cf/volumes" Jul 08 03:42:42.712895 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:42.712862 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jzq2l"] Jul 08 03:42:45.499166 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:45.499132 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mdlnj_90eb5f58-6d3f-4353-8d61-d6cf7891efed/manager/0.log" Jul 08 03:42:45.840204 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:45.840106 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vgv8m_41e60c26-8066-4069-9db0-7ddbecf277f1/server/0.log" Jul 08 03:42:45.956522 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:45.956466 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-45dln_4604d641-e2b0-4c88-9428-a8cec39030e4/manager/0.log" Jul 08 03:42:46.186888 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:46.186864 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64b8ffddd6-72rc4_28b83991-6db1-4ebf-8052-e081b7e91def/copy-manifests/0.log" Jul 08 03:42:46.200417 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:46.200375 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64b8ffddd6-72rc4_28b83991-6db1-4ebf-8052-e081b7e91def/manager/0.log" Jul 08 03:42:47.699404 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:47.699342 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7fb2t_bfbddb3e-5995-43a3-a7b5-1bbdd3047763/manager/0.log" Jul 08 03:42:47.806831 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:47.806778 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-vxtpv_09eaec25-bde3-4702-badd-b132f6d7dc2c/manager/0.log" Jul 08 03:42:48.027669 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:48.027583 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-sgp7l_d1d03998-4db4-4eaa-b0ae-8972c76cbaa9/registry-server/0.log" Jul 08 03:42:48.143460 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:48.143428 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dfxcd_07a3cb3f-6867-4fba-b0c5-9f3294233ca6/manager/0.log" Jul 08 03:42:48.248218 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:48.248185 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jzq2l_05db8d30-1a63-44fc-af00-baac2f1825bf/limitador/0.log" Jul 08 03:42:48.362783 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:48.362710 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-mtxfm_5493b764-e3f9-4f06-9109-13a1543efb8b/manager/0.log" Jul 08 03:42:49.035210 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:49.035183 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b85c4fb4-kwjfj_f2908c98-8422-474a-97d4-95f5e41cdbe1/kube-auth-proxy/0.log" Jul 08 03:42:49.495235 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:49.495202 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c46bc767-r4rqj_3a708c11-6608-4fd6-8f47-f2d9ded1569a/router/0.log" Jul 08 03:42:50.040389 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:50.040350 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-h5mqv_afa37267-37b4-4518-97b1-9d8b3be28cb6/main/0.log" Jul 08 03:42:54.190877 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.190798 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l26gf/must-gather-wwcfb"] Jul 08 03:42:54.191250 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.191227 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" containerName="istio-proxy" Jul 08 03:42:54.191250 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.191247 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" containerName="istio-proxy" Jul 08 03:42:54.191331 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.191260 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" containerName="istio-proxy" Jul 08 03:42:54.191331 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.191266 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" containerName="istio-proxy" Jul 08 03:42:54.191331 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.191279 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" containerName="istio-proxy" Jul 08 03:42:54.191331 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.191286 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" containerName="istio-proxy" Jul 08 03:42:54.191479 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.191393 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" containerName="istio-proxy" Jul 08 03:42:54.191479 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.191404 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" containerName="istio-proxy" Jul 08 03:42:54.191479 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.191413 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cabf3e4-1be3-4b45-bfe2-f547e3e069cf" containerName="istio-proxy" Jul 08 03:42:54.191479 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.191418 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" containerName="istio-proxy" Jul 08 03:42:54.191624 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.191509 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" containerName="istio-proxy" Jul 08 03:42:54.191624 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.191519 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a21b287b-e9a6-4aa2-bdbe-d08b24308fbf" containerName="istio-proxy" Jul 08 03:42:54.196214 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.196198 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l26gf/must-gather-wwcfb" Jul 08 03:42:54.198841 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.198817 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l26gf\"/\"openshift-service-ca.crt\"" Jul 08 03:42:54.198982 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.198846 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l26gf\"/\"kube-root-ca.crt\"" Jul 08 03:42:54.198982 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.198815 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96879d5f-58c9-4725-bebc-3710cfb8093d-must-gather-output\") pod \"must-gather-wwcfb\" (UID: \"96879d5f-58c9-4725-bebc-3710cfb8093d\") " pod="openshift-must-gather-l26gf/must-gather-wwcfb" Jul 08 03:42:54.198982 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.198938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-628xd\" (UniqueName: \"kubernetes.io/projected/96879d5f-58c9-4725-bebc-3710cfb8093d-kube-api-access-628xd\") pod \"must-gather-wwcfb\" (UID: \"96879d5f-58c9-4725-bebc-3710cfb8093d\") " pod="openshift-must-gather-l26gf/must-gather-wwcfb" Jul 08 03:42:54.200310 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.200281 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l26gf\"/\"default-dockercfg-x5rmv\"" Jul 08 03:42:54.202409 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.202384 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l26gf/must-gather-wwcfb"] Jul 08 03:42:54.299858 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.299825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-628xd\" (UniqueName: \"kubernetes.io/projected/96879d5f-58c9-4725-bebc-3710cfb8093d-kube-api-access-628xd\") pod \"must-gather-wwcfb\" (UID: \"96879d5f-58c9-4725-bebc-3710cfb8093d\") " pod="openshift-must-gather-l26gf/must-gather-wwcfb" Jul 08 03:42:54.300056 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.299886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96879d5f-58c9-4725-bebc-3710cfb8093d-must-gather-output\") pod \"must-gather-wwcfb\" (UID: \"96879d5f-58c9-4725-bebc-3710cfb8093d\") " pod="openshift-must-gather-l26gf/must-gather-wwcfb" Jul 08 03:42:54.300195 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.300178 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96879d5f-58c9-4725-bebc-3710cfb8093d-must-gather-output\") pod \"must-gather-wwcfb\" (UID: \"96879d5f-58c9-4725-bebc-3710cfb8093d\") " pod="openshift-must-gather-l26gf/must-gather-wwcfb" Jul 08 03:42:54.311999 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.311975 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-628xd\" (UniqueName: \"kubernetes.io/projected/96879d5f-58c9-4725-bebc-3710cfb8093d-kube-api-access-628xd\") pod \"must-gather-wwcfb\" (UID: \"96879d5f-58c9-4725-bebc-3710cfb8093d\") " pod="openshift-must-gather-l26gf/must-gather-wwcfb" Jul 08 03:42:54.507075 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.506967 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l26gf/must-gather-wwcfb" Jul 08 03:42:54.644664 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.644640 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l26gf/must-gather-wwcfb"] Jul 08 03:42:54.647064 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:42:54.647016 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96879d5f_58c9_4725_bebc_3710cfb8093d.slice/crio-37fcce2ef7995e309482dd74209d65330d98eb2be2e017c6e74eb46ec15a9ff1 WatchSource:0}: Error finding container 37fcce2ef7995e309482dd74209d65330d98eb2be2e017c6e74eb46ec15a9ff1: Status 404 returned error can't find the container with id 37fcce2ef7995e309482dd74209d65330d98eb2be2e017c6e74eb46ec15a9ff1 Jul 08 03:42:54.649096 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.649075 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 03:42:54.983226 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:54.983189 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l26gf/must-gather-wwcfb" event={"ID":"96879d5f-58c9-4725-bebc-3710cfb8093d","Type":"ContainerStarted","Data":"37fcce2ef7995e309482dd74209d65330d98eb2be2e017c6e74eb46ec15a9ff1"} Jul 08 03:42:55.993814 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:55.992896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l26gf/must-gather-wwcfb" event={"ID":"96879d5f-58c9-4725-bebc-3710cfb8093d","Type":"ContainerStarted","Data":"16ea692927211161f7fe482b7ccfef266ad5071faa8bae49ecd836ea05915dee"} Jul 08 03:42:55.993814 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:55.992955 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l26gf/must-gather-wwcfb" event={"ID":"96879d5f-58c9-4725-bebc-3710cfb8093d","Type":"ContainerStarted","Data":"10d7b9f9e893fe53330a028bcc7b23568f3a35fab557c0954c504b84eb41541e"} Jul 08 03:42:56.010697 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:56.010641 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l26gf/must-gather-wwcfb" podStartSLOduration=1.108591519 podStartE2EDuration="2.010624722s" totalImagesPullingTime="902.033203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:42:56.008966328 +0000 UTC m=+2694.999414377" watchObservedRunningTime="2026-07-08 03:42:56.010624722 +0000 UTC m=+2695.001072759" Jul 08 03:42:57.112968 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:57.112938 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-78f5d_0343e166-1101-45b0-a57e-cc26b4858985/global-pull-secret-syncer/0.log" Jul 08 03:42:57.232496 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:57.232461 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9sj9r_4e25aa2e-b99d-492f-abb2-46c3c8fc0662/konnectivity-agent/0.log" Jul 08 03:42:57.323116 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:42:57.323028 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-99.ec2.internal_8b8d67d3d75c39b81a69b1a068c27f3b/haproxy/0.log" Jul 08 03:43:01.714354 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:01.714321 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:43:01.715897 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:01.715874 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:43:01.729745 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:01.729713 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:43:01.730380 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:01.730358 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:43:02.405511 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:02.405481 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7fb2t_bfbddb3e-5995-43a3-a7b5-1bbdd3047763/manager/0.log" Jul 08 03:43:02.436016 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:02.435977 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-vxtpv_09eaec25-bde3-4702-badd-b132f6d7dc2c/manager/0.log" Jul 08 03:43:02.506951 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:02.506919 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-sgp7l_d1d03998-4db4-4eaa-b0ae-8972c76cbaa9/registry-server/0.log" Jul 08 03:43:02.686700 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:02.686661 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dfxcd_07a3cb3f-6867-4fba-b0c5-9f3294233ca6/manager/0.log" Jul 08 03:43:02.710728 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:02.710692 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jzq2l_05db8d30-1a63-44fc-af00-baac2f1825bf/limitador/0.log" Jul 08 03:43:02.786774 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:02.786741 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-mtxfm_5493b764-e3f9-4f06-9109-13a1543efb8b/manager/0.log" Jul 08 03:43:04.411516 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:04.411470 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-pqdv5_cce8a6b5-0ecb-4ce6-bad9-e4387039a932/cluster-monitoring-operator/0.log" Jul 08 03:43:04.729481 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:04.729386 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5h8n_f7ae4070-9290-4092-bc23-7e1ce413daff/node-exporter/0.log" Jul 08 03:43:04.747024 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:04.746995 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5h8n_f7ae4070-9290-4092-bc23-7e1ce413daff/kube-rbac-proxy/0.log" Jul 08 03:43:04.769009 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:04.768972 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5h8n_f7ae4070-9290-4092-bc23-7e1ce413daff/init-textfile/0.log" Jul 08 03:43:05.164924 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:05.164853 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-64bj8_e78201e6-cbb1-4203-a1f4-103c40ceb783/prometheus-operator-admission-webhook/0.log" Jul 08 03:43:06.682567 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:06.682533 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-rrkzw_5a53b345-6e70-44ee-b799-4f4c24c1d094/networking-console-plugin/0.log" Jul 08 03:43:07.106766 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.106684 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt"] Jul 08 03:43:07.111835 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.111803 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.118483 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.118445 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt"] Jul 08 03:43:07.226174 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.226132 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6653005c-f72b-4d7d-bf6f-57e166c7f43f-sys\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.226461 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.226434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6653005c-f72b-4d7d-bf6f-57e166c7f43f-lib-modules\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.226642 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.226624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6653005c-f72b-4d7d-bf6f-57e166c7f43f-proc\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.226804 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.226788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6653005c-f72b-4d7d-bf6f-57e166c7f43f-podres\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.227023 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.227005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cq87\" (UniqueName: \"kubernetes.io/projected/6653005c-f72b-4d7d-bf6f-57e166c7f43f-kube-api-access-6cq87\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.323319 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.323263 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/1.log" Jul 08 03:43:07.327653 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.327625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6653005c-f72b-4d7d-bf6f-57e166c7f43f-sys\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.327653 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.327672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6653005c-f72b-4d7d-bf6f-57e166c7f43f-lib-modules\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.327995 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.327726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6653005c-f72b-4d7d-bf6f-57e166c7f43f-proc\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.327995 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.327751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6653005c-f72b-4d7d-bf6f-57e166c7f43f-podres\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.327995 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.327759 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6653005c-f72b-4d7d-bf6f-57e166c7f43f-sys\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.327995 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.327853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cq87\" (UniqueName: \"kubernetes.io/projected/6653005c-f72b-4d7d-bf6f-57e166c7f43f-kube-api-access-6cq87\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.327995 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.327859 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6653005c-f72b-4d7d-bf6f-57e166c7f43f-proc\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.327995 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.327877 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6653005c-f72b-4d7d-bf6f-57e166c7f43f-podres\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.328487 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.327884 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6653005c-f72b-4d7d-bf6f-57e166c7f43f-lib-modules\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.330964 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.330943 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lrzh5_d4220e41-4e35-4cc8-a67b-db04dd99e5cb/console-operator/2.log" Jul 08 03:43:07.337726 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.337697 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cq87\" (UniqueName: \"kubernetes.io/projected/6653005c-f72b-4d7d-bf6f-57e166c7f43f-kube-api-access-6cq87\") pod \"perf-node-gather-daemonset-2jnlt\" (UID: \"6653005c-f72b-4d7d-bf6f-57e166c7f43f\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.426220 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.426180 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:07.645142 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.645112 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt"] Jul 08 03:43:07.647570 ip-10-0-142-99 kubenswrapper[2576]: W0708 03:43:07.647515 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6653005c_f72b_4d7d_bf6f_57e166c7f43f.slice/crio-c4383bcab9cfa98f26a6007f8a407078028f727ec905f79ecb00b680d9fe904d WatchSource:0}: Error finding container c4383bcab9cfa98f26a6007f8a407078028f727ec905f79ecb00b680d9fe904d: Status 404 returned error can't find the container with id c4383bcab9cfa98f26a6007f8a407078028f727ec905f79ecb00b680d9fe904d Jul 08 03:43:07.919604 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.919576 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-565c746b98-6n4g2_04c9ac21-d891-4dbc-86d9-0b236ef06072/console/0.log" Jul 08 03:43:07.952432 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:07.952349 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-cgq8z_819aac06-7339-40a7-b54d-cffb09e62092/download-server/0.log" Jul 08 03:43:08.092075 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:08.092022 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" event={"ID":"6653005c-f72b-4d7d-bf6f-57e166c7f43f","Type":"ContainerStarted","Data":"ab605cae207b08ccdccf17079ac34139ee64e2a29ba2597a27fdd7dd99ec0c53"} Jul 08 03:43:08.092246 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:08.092103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" event={"ID":"6653005c-f72b-4d7d-bf6f-57e166c7f43f","Type":"ContainerStarted","Data":"c4383bcab9cfa98f26a6007f8a407078028f727ec905f79ecb00b680d9fe904d"} Jul 08 03:43:08.092246 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:08.092166 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:08.110443 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:08.110396 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" podStartSLOduration=1.110381132 podStartE2EDuration="1.110381132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 03:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:43:08.108231798 +0000 UTC m=+2707.098679848" watchObservedRunningTime="2026-07-08 03:43:08.110381132 +0000 UTC m=+2707.100829168" Jul 08 03:43:08.537251 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:08.537198 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-cfrk8_df234fb8-8adb-4814-836d-7dfb0ea5a135/volume-data-source-validator/0.log" Jul 08 03:43:09.541156 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:09.541129 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6j82n_94c47594-a484-4654-8cae-7325c0d9d581/dns/0.log" Jul 08 03:43:09.559631 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:09.559607 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6j82n_94c47594-a484-4654-8cae-7325c0d9d581/kube-rbac-proxy/0.log" Jul 08 03:43:09.646418 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:09.646392 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h69hf_b53a5c28-a70a-4388-bb82-03cd3832d27a/dns-node-resolver/0.log" Jul 08 03:43:10.176924 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:10.176894 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-57xjh_97e0ac5b-7e47-49c6-a320-6f45f632c3d4/node-ca/0.log" Jul 08 03:43:11.285615 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:11.285581 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b85c4fb4-kwjfj_f2908c98-8422-474a-97d4-95f5e41cdbe1/kube-auth-proxy/0.log" Jul 08 03:43:11.448227 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:11.448165 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c46bc767-r4rqj_3a708c11-6608-4fd6-8f47-f2d9ded1569a/router/0.log" Jul 08 03:43:12.195039 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:12.195004 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lbwvc_cd9ca7be-62a3-4895-8e9c-ecaa09ea7f03/serve-healthcheck-canary/0.log" Jul 08 03:43:12.645084 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:12.644982 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-jz8tg_f73570e2-30eb-4cb3-9d67-0b6689c9f312/insights-operator/0.log" Jul 08 03:43:12.645673 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:12.645655 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-jz8tg_f73570e2-30eb-4cb3-9d67-0b6689c9f312/insights-operator/1.log" Jul 08 03:43:12.793151 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:12.793120 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rz2g4_5c7e8542-42da-4659-bccb-1b8bee7ec4d6/kube-rbac-proxy/0.log" Jul 08 03:43:12.810909 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:12.810884 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rz2g4_5c7e8542-42da-4659-bccb-1b8bee7ec4d6/exporter/0.log" Jul 08 03:43:12.829012 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:12.828980 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rz2g4_5c7e8542-42da-4659-bccb-1b8bee7ec4d6/extractor/0.log" Jul 08 03:43:14.114411 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:14.114355 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-2jnlt" Jul 08 03:43:14.926010 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:14.925979 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mdlnj_90eb5f58-6d3f-4353-8d61-d6cf7891efed/manager/0.log" Jul 08 03:43:15.139841 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:15.139802 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vgv8m_41e60c26-8066-4069-9db0-7ddbecf277f1/server/0.log" Jul 08 03:43:15.217569 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:15.217444 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-45dln_4604d641-e2b0-4c88-9428-a8cec39030e4/manager/0.log" Jul 08 03:43:15.356106 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:15.356038 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64b8ffddd6-72rc4_28b83991-6db1-4ebf-8052-e081b7e91def/manager/0.log" Jul 08 03:43:15.371617 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:15.371587 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64b8ffddd6-72rc4_28b83991-6db1-4ebf-8052-e081b7e91def/copy-manifests/0.log" Jul 08 03:43:17.930355 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:17.930326 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b85c4fb4-kwjfj_f2908c98-8422-474a-97d4-95f5e41cdbe1/kube-auth-proxy/0.log" Jul 08 03:43:18.055065 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:18.055024 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c46bc767-r4rqj_3a708c11-6608-4fd6-8f47-f2d9ded1569a/router/0.log" Jul 08 03:43:19.418216 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:19.418188 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b85c4fb4-kwjfj_f2908c98-8422-474a-97d4-95f5e41cdbe1/kube-auth-proxy/0.log" Jul 08 03:43:19.545722 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:19.545688 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c46bc767-r4rqj_3a708c11-6608-4fd6-8f47-f2d9ded1569a/router/0.log" Jul 08 03:43:21.959141 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:21.959092 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-sfp2b_7d9a4a38-9c77-49b5-b1e1-9f423d2e8bc8/migrator/0.log" Jul 08 03:43:21.976489 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:21.976459 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-sfp2b_7d9a4a38-9c77-49b5-b1e1-9f423d2e8bc8/graceful-termination/0.log" Jul 08 03:43:22.310154 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:22.310039 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-45htx_685a7232-5ff7-4f67-b949-1c41b98bb3c3/kube-storage-version-migrator-operator/1.log" Jul 08 03:43:22.311198 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:22.311175 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-45htx_685a7232-5ff7-4f67-b949-1c41b98bb3c3/kube-storage-version-migrator-operator/0.log" Jul 08 03:43:23.554373 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:23.554348 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nf95s_ab087428-a2bf-4ca5-8c12-1824860547c2/kube-multus-additional-cni-plugins/0.log" Jul 08 03:43:23.574307 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:23.574279 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nf95s_ab087428-a2bf-4ca5-8c12-1824860547c2/egress-router-binary-copy/0.log" Jul 08 03:43:23.591401 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:23.591379 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nf95s_ab087428-a2bf-4ca5-8c12-1824860547c2/cni-plugins/0.log" Jul 08 03:43:23.608341 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:23.608319 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nf95s_ab087428-a2bf-4ca5-8c12-1824860547c2/bond-cni-plugin/0.log" Jul 08 03:43:23.628684 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:23.628655 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nf95s_ab087428-a2bf-4ca5-8c12-1824860547c2/routeoverride-cni/0.log" Jul 08 03:43:23.645957 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:23.645929 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nf95s_ab087428-a2bf-4ca5-8c12-1824860547c2/whereabouts-cni-bincopy/0.log" Jul 08 03:43:23.663447 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:23.663424 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nf95s_ab087428-a2bf-4ca5-8c12-1824860547c2/whereabouts-cni/0.log" Jul 08 03:43:23.747559 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:23.747526 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-q2md7_97edc872-75d9-449b-9fa3-0871dc3a965e/kube-multus/0.log" Jul 08 03:43:23.804333 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:23.804304 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wdvzz_c392372d-4d45-4636-8058-57fd87930e8e/network-metrics-daemon/0.log" Jul 08 03:43:23.820522 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:23.820448 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wdvzz_c392372d-4d45-4636-8058-57fd87930e8e/kube-rbac-proxy/0.log" Jul 08 03:43:24.703974 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:24.703943 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-controller/0.log" Jul 08 03:43:24.718233 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:24.718207 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/1.log" Jul 08 03:43:24.718448 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:24.718212 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/ovn-acl-logging/0.log" Jul 08 03:43:24.738551 ip-10-0-142-99 kubenswrapper[2576]: I0708 03:43:24.738526 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qb87_3bdbbb3c-a277-4cb9-9d63-d626cc52142e/kube-rbac-proxy-node/0.log"