Jul 22 16:05:09.295728 ip-10-0-131-21 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 16:05:09.295739 ip-10-0-131-21 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 16:05:09.295745 ip-10-0-131-21 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 16:05:09.295966 ip-10-0-131-21 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 16:05:19.366723 ip-10-0-131-21 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 16:05:19.366741 ip-10-0-131-21 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 97737cb7f1a144d7bc119951a07f7d09 -- Jul 22 16:07:13.653837 ip-10-0-131-21 systemd[1]: Starting Kubernetes Kubelet... Jul 22 16:07:15.564781 ip-10-0-131-21 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:07:15.564781 ip-10-0-131-21 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 16:07:15.564781 ip-10-0-131-21 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:07:15.564781 ip-10-0-131-21 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 16:07:15.564781 ip-10-0-131-21 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:07:15.564781 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.549301 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 16:07:15.564781 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560304 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:07:15.564781 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560319 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:07:15.564781 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560323 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:07:15.564781 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560328 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:07:15.564781 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560333 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:07:15.564781 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560336 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:07:15.564781 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560339 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:07:15.564781 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560342 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560345 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560348 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560350 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560353 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560356 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560359 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560361 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560364 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560366 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560376 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560379 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560382 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560384 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560387 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560389 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560392 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560394 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560397 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560399 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:07:15.565568 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560401 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560404 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560407 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560410 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560412 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560415 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560418 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560420 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560423 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560425 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560428 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560431 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560433 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560436 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560438 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560441 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560443 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560447 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560451 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:07:15.566398 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560455 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560458 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560460 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560463 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560467 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560470 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560472 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560474 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560477 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560479 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560482 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560484 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560487 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560489 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560492 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560494 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560497 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560500 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560502 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560505 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:07:15.566997 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560508 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560511 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560513 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560516 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560518 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560521 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560523 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560526 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560528 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560530 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560533 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560535 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560538 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560540 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560542 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560545 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560547 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560550 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560553 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560555 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:07:15.567520 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560558 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560561 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560563 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560565 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560568 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560570 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560573 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560575 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560579 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560582 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560585 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560587 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.560590 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561287 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561294 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561297 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561300 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561303 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561306 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561309 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:07:15.568166 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561312 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561315 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561318 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561321 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561324 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561327 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561330 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561332 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561335 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561337 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561340 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561343 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561346 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561348 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561351 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561354 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561358 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561360 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561363 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561365 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:07:15.568830 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561368 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561370 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561373 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561376 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561378 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561381 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561384 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561387 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561389 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561392 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561394 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561397 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561399 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561402 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561404 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561407 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561410 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561412 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561415 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561417 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:07:15.569546 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561420 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561422 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561424 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561427 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561430 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561433 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561435 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561438 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561440 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561443 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561445 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561448 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561450 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561453 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561456 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561458 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561461 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561464 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561467 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:07:15.570073 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561470 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561472 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561477 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561480 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561483 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561486 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561489 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561492 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561494 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561496 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561499 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561504 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561507 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561510 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561513 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561516 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561519 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561522 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561525 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:07:15.676134 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561528 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:07:15.641722 ip-10-0-131-21 systemd[1]: Started Kubernetes Kubelet. Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561531 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561533 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561536 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561538 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561541 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561543 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561545 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561548 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561551 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561554 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561556 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561559 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561561 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561944 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561949 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561952 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561955 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561958 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561962 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:07:15.677330 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561964 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561967 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561969 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561972 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561974 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561977 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561979 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561982 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561985 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561990 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561994 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561996 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.561999 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562002 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562006 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562008 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562011 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562013 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562016 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562018 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:07:15.685617 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562021 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562024 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562028 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562031 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562034 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562037 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562040 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562042 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562045 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562047 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562050 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562052 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562055 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562057 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562060 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562062 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562064 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562068 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562070 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:07:15.686780 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562073 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562076 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562079 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562082 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562085 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562088 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562090 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562093 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562095 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562098 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562100 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562103 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562105 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562108 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562111 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562113 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562116 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562118 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562121 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562123 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:07:15.687538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562126 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562129 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562131 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562134 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562136 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562139 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562142 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562144 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562146 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562149 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562152 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562155 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562158 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562161 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562165 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562167 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562170 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562173 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562175 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562178 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:07:15.779508 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562180 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562182 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562186 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562189 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562193 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562196 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562199 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562202 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562205 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562207 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562210 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562212 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562215 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562218 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562286 2567 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562292 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562306 2567 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562313 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562318 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562323 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562328 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 16:07:15.780257 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562332 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562335 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562338 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562341 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562345 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562349 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562352 2567 flags.go:64] FLAG: --cgroup-root="" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562355 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562358 2567 flags.go:64] FLAG: --client-ca-file="" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562362 2567 flags.go:64] FLAG: --cloud-provider="external" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562365 2567 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562369 2567 flags.go:64] FLAG: --cluster-domain="" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562371 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562374 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562377 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562381 2567 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562385 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562388 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562391 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562394 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562397 2567 flags.go:64] FLAG: --contention-profiling="false" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562399 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562402 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562405 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 16:07:15.781367 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562408 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562412 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562415 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562418 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562420 2567 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562423 2567 flags.go:64] FLAG: --enable-server="true" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562426 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562430 2567 flags.go:64] FLAG: --event-burst="100" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562433 2567 flags.go:64] FLAG: --event-qps="50" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562436 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562440 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562443 2567 flags.go:64] FLAG: --eviction-hard="" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562447 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562451 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562454 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562457 2567 flags.go:64] FLAG: --eviction-soft="" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562459 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562462 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562466 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562469 2567 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562471 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562474 2567 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562477 2567 flags.go:64] FLAG: --feature-gates="" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562480 2567 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562483 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 16:07:15.782147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562486 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562489 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562492 2567 flags.go:64] FLAG: --healthz-port="10248" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562495 2567 flags.go:64] FLAG: --help="false" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562497 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-131-21.ec2.internal" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562500 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562503 2567 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562506 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562509 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562512 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562515 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562519 2567 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562522 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562525 2567 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562527 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562531 2567 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562534 2567 flags.go:64] FLAG: --kube-reserved="" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562537 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562539 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562542 2567 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562546 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562549 2567 flags.go:64] FLAG: --lock-file="" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562552 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562554 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 16:07:15.783646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562557 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562562 2567 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562568 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562571 2567 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562573 2567 flags.go:64] FLAG: --logging-format="text" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562576 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562579 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562582 2567 flags.go:64] FLAG: --manifest-url="" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562584 2567 flags.go:64] FLAG: --manifest-url-header="" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562594 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562597 2567 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562605 2567 flags.go:64] FLAG: --max-pods="110" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562608 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562611 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562614 2567 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562616 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562619 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562622 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562625 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562633 2567 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562636 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562639 2567 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562642 2567 flags.go:64] FLAG: --pod-cidr="" Jul 22 16:07:15.785920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562645 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562649 2567 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562652 2567 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562655 2567 flags.go:64] FLAG: --pods-per-core="0" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562658 2567 flags.go:64] FLAG: --port="10250" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562662 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562665 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-094b32b14d2bb844a" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562668 2567 flags.go:64] FLAG: --qos-reserved="" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562671 2567 flags.go:64] FLAG: --read-only-port="10255" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562674 2567 flags.go:64] FLAG: --register-node="true" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562677 2567 flags.go:64] FLAG: --register-with-taints="" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562681 2567 flags.go:64] FLAG: --registry-burst="10" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562684 2567 flags.go:64] FLAG: --registry-qps="5" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562699 2567 flags.go:64] FLAG: --reserved-cpus="" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562702 2567 flags.go:64] FLAG: --reserved-memory="" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562705 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562709 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562711 2567 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562714 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562717 2567 flags.go:64] FLAG: --runonce="false" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562720 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562723 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562726 2567 flags.go:64] FLAG: --seccomp-default="false" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562728 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562731 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562734 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 16:07:15.786670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562737 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562740 2567 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562742 2567 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562746 2567 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562749 2567 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562752 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562755 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562758 2567 flags.go:64] FLAG: --system-cgroups="" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562760 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562766 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562768 2567 flags.go:64] FLAG: --tls-cert-file="" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562772 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562776 2567 flags.go:64] FLAG: --tls-min-version="" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562779 2567 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562782 2567 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562784 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562787 2567 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562790 2567 flags.go:64] FLAG: --v="2" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562793 2567 flags.go:64] FLAG: --version="false" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562797 2567 flags.go:64] FLAG: --vmodule="" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562801 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.562804 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562917 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562921 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:07:15.787435 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562924 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562927 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562929 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562932 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562934 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562937 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562939 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562942 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562944 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562946 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562949 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562953 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562955 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562958 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562961 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562963 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562965 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562968 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562970 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562974 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:07:15.788154 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562977 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562979 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562982 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562984 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562986 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562989 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562991 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562994 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.562997 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563000 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563002 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563005 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563007 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563010 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563012 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563015 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563017 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563020 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563022 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563025 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:07:15.788761 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563027 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563030 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563032 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563036 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563038 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563041 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563043 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563046 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563048 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563051 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563053 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563057 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563060 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563062 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563065 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563068 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563070 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563073 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563075 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563078 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:07:15.789409 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563080 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563083 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563085 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563088 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563090 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563093 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563095 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563098 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563100 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563103 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563106 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563111 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563114 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563119 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563122 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563126 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563129 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563132 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563135 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:07:15.789998 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563138 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563140 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563143 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563146 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563149 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563152 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563154 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563157 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563159 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563162 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563164 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563167 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563169 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563172 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563174 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563177 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563180 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:07:15.790547 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.563183 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:07:15.791061 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.563710 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:07:15.791061 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.569996 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 16:07:15.791061 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.570011 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 16:07:15.791061 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.570633 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 16:07:15.791061 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.570640 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 16:07:15.791061 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570706 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:07:15.791061 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570711 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:07:15.791061 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570715 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:07:15.791061 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570719 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:07:15.791061 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570722 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:07:15.791061 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570725 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:07:15.791061 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570727 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:07:15.791061 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570730 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:07:15.791061 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570733 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570735 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570738 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570740 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570743 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570746 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570749 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570751 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570753 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570756 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570759 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570761 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570764 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570767 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570770 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570773 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570775 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570778 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570780 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570783 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:07:15.791482 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570785 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570789 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570791 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570794 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570796 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570798 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570801 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570803 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570806 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570810 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570814 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570817 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570819 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570822 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570825 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570827 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570830 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570832 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570835 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:07:15.792072 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570838 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570840 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570843 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570845 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570848 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570850 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570853 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570855 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570858 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570861 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570864 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570866 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570869 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570872 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570875 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570878 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570881 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570883 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570886 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570888 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:07:15.792623 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570891 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570894 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570896 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570899 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570901 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570904 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570906 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570909 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570912 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570915 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570918 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570921 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570924 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570927 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570930 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570932 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570935 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570938 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570940 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:07:15.793350 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570943 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570946 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570949 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570951 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570954 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570957 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570959 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570962 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570966 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570969 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570972 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570975 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.570978 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.570983 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571091 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:07:15.793936 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571096 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571099 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571102 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571104 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571107 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571110 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571112 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571115 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571118 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571120 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571123 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571125 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571128 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571130 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571133 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571135 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571138 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571140 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571144 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571148 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:07:15.794379 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571151 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571154 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571157 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571160 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571162 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571165 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571167 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571170 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571189 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571193 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571196 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571199 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571202 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571204 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571207 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571209 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571212 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571214 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571217 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571220 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:07:15.794981 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571222 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571225 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571227 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571230 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571233 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571237 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571240 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571243 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571245 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571248 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571250 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571253 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571256 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571259 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571261 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571264 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571267 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571269 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571272 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:07:15.795550 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571274 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571277 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571280 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571283 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571285 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571288 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571290 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571293 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571296 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571300 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571303 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571305 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571308 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571310 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571313 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571315 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571318 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571320 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571323 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:07:15.796156 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571326 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571328 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571331 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571334 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571336 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571339 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571342 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571344 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571347 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571349 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571352 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571355 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571357 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571359 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571362 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571366 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571368 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571371 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571375 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:07:15.796723 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:15.571377 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:07:15.797292 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.571382 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:07:15.797292 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.571480 2567 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 16:07:15.797292 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.573151 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 16:07:15.797292 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.574178 2567 server.go:1019] "Starting client certificate rotation" Jul 22 16:07:15.797292 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.574279 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 16:07:15.797292 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.574314 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 16:07:15.797292 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.576851 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 16:07:15.797292 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.589441 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 16:07:15.797292 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.604761 2567 log.go:25] "Validated CRI v1 runtime API" Jul 22 16:07:15.797292 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.606571 2567 log.go:25] "Validated CRI v1 image API" Jul 22 16:07:15.797292 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.608059 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 16:07:15.797292 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.617470 2567 fs.go:135] Filesystem UUIDs: map[4dbc6bff-6363-4f98-9852-c95797460541:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b9ca86de-bf62-4996-afe1-b4c7cf018873:/dev/nvme0n1p3] Jul 22 16:07:15.797666 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.617485 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 16:07:15.797666 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.620400 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.622786 2567 manager.go:216] Machine: {Timestamp:2026-07-22 16:07:15.621667627 +0000 UTC m=+0.763541714 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3196465 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec280593ba7dda67f4c1910944c0d20e SystemUUID:ec280593-ba7d-da67-f4c1-910944c0d20e BootID:97737cb7-f1a1-44d7-bc11-9951a07f7d09 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a2:dd:ef:d4:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a2:dd:ef:d4:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:85:b3:ec:f6:1c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.622887 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.623004 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.625156 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.625185 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-21.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.626354 2567 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.626364 2567 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.626371 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.626874 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.627797 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.631092 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.631194 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.631216 2567 kubelet.go:486] "Attempting to sync node with API server" Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.631225 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.631238 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.631248 2567 kubelet.go:398] "Adding apiserver pod source" Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.631255 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 16:07:15.797748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.631745 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.631776 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.633516 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.635117 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.635144 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.636133 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.636145 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.636151 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.636158 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.636166 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.636171 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.636184 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.636189 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.636195 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.636200 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.636211 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.636217 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.637130 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.637137 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.640817 2567 server.go:1268] "Started kubelet" Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.642043 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.642094 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 16:07:15.798322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.642755 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 16:07:15.805085 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:15.805045 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 16:07:15.805212 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.805064 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5wz77" Jul 22 16:07:15.805212 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.805111 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5wz77" Jul 22 16:07:15.805212 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.805195 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-21.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 16:07:15.805443 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:15.805409 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-21.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 16:07:15.805839 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.805816 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 16:07:15.807068 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:15.807044 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 16:07:15.807226 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.807213 2567 server.go:310] "Adding debug handlers to kubelet server" Jul 22 16:07:15.810005 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:15.809086 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-21.ec2.internal.18c4a855101022a5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-21.ec2.internal,UID:ip-10-0-131-21.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-21.ec2.internal,},FirstTimestamp:2026-07-22 16:07:15.640738469 +0000 UTC m=+0.782612617,LastTimestamp:2026-07-22 16:07:15.640738469 +0000 UTC m=+0.782612617,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-21.ec2.internal,}" Jul 22 16:07:15.812877 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.812856 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 16:07:15.812877 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.812868 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 16:07:15.813025 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:15.812925 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:15.813281 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.813267 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 16:07:15.813364 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.813270 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 16:07:15.813364 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.813300 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 16:07:15.813452 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.813378 2567 reconstruct.go:97] "Volume reconstruction finished" Jul 22 16:07:15.813452 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.813388 2567 reconciler.go:29] "Reconciler: start to sync state" Jul 22 16:07:15.813782 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.813765 2567 factory.go:55] Registering systemd factory Jul 22 16:07:15.813782 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.813782 2567 factory.go:223] Registration of the systemd container factory successfully Jul 22 16:07:15.814012 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.813994 2567 factory.go:153] Registering CRI-O factory Jul 22 16:07:15.814012 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.814013 2567 factory.go:223] Registration of the crio container factory successfully Jul 22 16:07:15.814118 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.814064 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 16:07:15.814118 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.814086 2567 factory.go:103] Registering Raw factory Jul 22 16:07:15.814118 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.814101 2567 manager.go:1196] Started watching for new ooms in manager Jul 22 16:07:15.815050 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.814991 2567 manager.go:349] Starting recovery of all containers Jul 22 16:07:15.815481 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.815181 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:07:15.818956 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:15.818932 2567 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-21.ec2.internal\" not found" node="ip-10-0-131-21.ec2.internal" Jul 22 16:07:15.825027 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.825010 2567 manager.go:354] Recovery completed Jul 22 16:07:15.828669 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:15.828648 2567 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 22 16:07:15.831502 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.831489 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:07:15.833986 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.833969 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:07:15.834074 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.833997 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:07:15.834074 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.834007 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:07:15.834532 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.834517 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 16:07:15.834598 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.834534 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 16:07:15.834598 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.834555 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 22 16:07:15.834598 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.834525 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:07:15.835353 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.835336 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:07:15.835419 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.835364 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:07:15.835419 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.835376 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:07:15.836730 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.836718 2567 policy_none.go:49] "None policy: Start" Jul 22 16:07:15.836770 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.836735 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 16:07:15.836770 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.836746 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 16:07:15.838644 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.838635 2567 policy_none.go:47] "Start" Jul 22 16:07:15.874413 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.874264 2567 manager.go:337] "Starting Device Plugin manager" Jul 22 16:07:15.874535 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:15.874438 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 16:07:15.874535 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.874451 2567 server.go:87] "Starting device plugin registration server" Jul 22 16:07:15.874713 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.874683 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 16:07:15.874759 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.874718 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 16:07:15.874831 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.874813 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 16:07:15.874911 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.874892 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 16:07:15.874911 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.874905 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 16:07:15.875394 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:15.875376 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 16:07:15.875479 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:15.875429 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:15.975397 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.975373 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:07:15.977309 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.975766 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 16:07:15.977309 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.976434 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:07:15.977309 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.976470 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:07:15.977309 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.976485 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:07:15.977309 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.976520 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-21.ec2.internal" Jul 22 16:07:15.977309 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.977161 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 16:07:15.977309 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.977222 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 16:07:15.977309 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.977267 2567 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 16:07:15.977309 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:15.977307 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 16:07:15.982854 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.982832 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:07:15.988790 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:15.988766 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-21.ec2.internal" Jul 22 16:07:15.988861 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:15.988805 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-21.ec2.internal\": node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:16.065756 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:16.065662 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:16.077726 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.077669 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal"] Jul 22 16:07:16.077820 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.077786 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:07:16.078789 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.078773 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:07:16.078871 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.078801 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:07:16.078871 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.078813 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:07:16.080887 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.080875 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:07:16.081021 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.081007 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.081063 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.081042 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:07:16.081564 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.081543 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:07:16.081612 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.081567 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:07:16.081612 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.081545 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:07:16.081612 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.081599 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:07:16.081612 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.081610 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:07:16.081732 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.081576 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:07:16.083557 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.083541 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.083605 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.083572 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:07:16.084255 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.084234 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:07:16.084255 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.084256 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:07:16.084345 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.084267 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:07:16.110215 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:16.110193 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-21.ec2.internal\" not found" node="ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.114337 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:16.114321 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-21.ec2.internal\" not found" node="ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.115378 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.115363 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59f68753e88fdefdb4f8aaf388e35bab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"59f68753e88fdefdb4f8aaf388e35bab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.115425 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.115389 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59f68753e88fdefdb4f8aaf388e35bab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"59f68753e88fdefdb4f8aaf388e35bab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.115425 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.115405 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f338029658e35a32ff779b9f1d620fb0-config\") pod \"kube-apiserver-proxy-ip-10-0-131-21.ec2.internal\" (UID: \"f338029658e35a32ff779b9f1d620fb0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.165756 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:16.165728 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:16.216345 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.216319 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59f68753e88fdefdb4f8aaf388e35bab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"59f68753e88fdefdb4f8aaf388e35bab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.216435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.216348 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59f68753e88fdefdb4f8aaf388e35bab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"59f68753e88fdefdb4f8aaf388e35bab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.216435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.216365 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f338029658e35a32ff779b9f1d620fb0-config\") pod \"kube-apiserver-proxy-ip-10-0-131-21.ec2.internal\" (UID: \"f338029658e35a32ff779b9f1d620fb0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.216435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.216404 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f338029658e35a32ff779b9f1d620fb0-config\") pod \"kube-apiserver-proxy-ip-10-0-131-21.ec2.internal\" (UID: \"f338029658e35a32ff779b9f1d620fb0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.216435 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.216425 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59f68753e88fdefdb4f8aaf388e35bab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"59f68753e88fdefdb4f8aaf388e35bab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.216564 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.216425 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59f68753e88fdefdb4f8aaf388e35bab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"59f68753e88fdefdb4f8aaf388e35bab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.266423 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:16.266394 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:16.366921 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:16.366866 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:16.413068 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.413044 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.417522 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.417434 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Jul 22 16:07:16.467850 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:16.467817 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:16.568288 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:16.568255 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:16.574414 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.574396 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 16:07:16.574561 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.574539 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 16:07:16.574596 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.574566 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 16:07:16.669218 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:16.669134 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:16.769726 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:16.769686 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:16.789857 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.789831 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:07:16.810704 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.810661 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 16:02:15 +0000 UTC" deadline="2028-03-19 02:32:37.797188748 +0000 UTC" Jul 22 16:07:16.810704 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.810703 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14530h25m20.986503073s" Jul 22 16:07:16.870092 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:16.870054 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:16.917647 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:16.917618 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f68753e88fdefdb4f8aaf388e35bab.slice/crio-c09933fb12b8a8442cc03e10be18c37780963332e8bedbe118efd2a45a565813 WatchSource:0}: Error finding container c09933fb12b8a8442cc03e10be18c37780963332e8bedbe118efd2a45a565813: Status 404 returned error can't find the container with id c09933fb12b8a8442cc03e10be18c37780963332e8bedbe118efd2a45a565813 Jul 22 16:07:16.919973 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:16.918525 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf338029658e35a32ff779b9f1d620fb0.slice/crio-3a91615caeb0e22381a17558fd97b417ca9f46dc06092b0c94def59b0643e945 WatchSource:0}: Error finding container 3a91615caeb0e22381a17558fd97b417ca9f46dc06092b0c94def59b0643e945: Status 404 returned error can't find the container with id 3a91615caeb0e22381a17558fd97b417ca9f46dc06092b0c94def59b0643e945 Jul 22 16:07:16.922840 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.922822 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:07:16.970773 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:16.970742 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:16.981196 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.981147 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" event={"ID":"59f68753e88fdefdb4f8aaf388e35bab","Type":"ContainerStarted","Data":"c09933fb12b8a8442cc03e10be18c37780963332e8bedbe118efd2a45a565813"} Jul 22 16:07:16.982097 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:16.982075 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" event={"ID":"f338029658e35a32ff779b9f1d620fb0","Type":"ContainerStarted","Data":"3a91615caeb0e22381a17558fd97b417ca9f46dc06092b0c94def59b0643e945"} Jul 22 16:07:17.071617 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.071585 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:17.172365 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.172293 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:17.272773 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.272739 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:17.373239 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.373203 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Jul 22 16:07:17.408162 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.408128 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:07:17.413422 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.413398 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Jul 22 16:07:17.428947 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.428881 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 16:07:17.429853 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.429827 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Jul 22 16:07:17.462408 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.462381 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 16:07:17.633005 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.632973 2567 apiserver.go:52] "Watching apiserver" Jul 22 16:07:17.639439 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.639413 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 16:07:17.640509 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.640481 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-vbpf9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv","openshift-cluster-node-tuning-operator/tuned-vjhsc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal","openshift-multus/multus-additional-cni-plugins-xq6wn","openshift-multus/network-metrics-daemon-6vr2t","openshift-network-diagnostics/network-check-target-rnnbk","openshift-network-operator/iptables-alerter-9xbmt","kube-system/global-pull-secret-syncer-f7rz8","kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal","openshift-dns/node-resolver-j8z9t","openshift-image-registry/node-ca-xspks","openshift-multus/multus-pr97d","openshift-ovn-kubernetes/ovnkube-node-g2pjb"] Jul 22 16:07:17.643066 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.642998 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.645762 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.645581 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.646677 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.646117 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 16:07:17.646677 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.646334 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 16:07:17.646677 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.646635 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 16:07:17.646941 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.646906 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-q49f7\"" Jul 22 16:07:17.647832 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.647099 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 16:07:17.649642 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.648008 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.649642 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.648357 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 16:07:17.650302 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.650281 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vbpf9" Jul 22 16:07:17.651120 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.650730 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 16:07:17.651120 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.650766 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 16:07:17.651120 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.650981 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-99sks\"" Jul 22 16:07:17.651120 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.651061 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qsrjk\"" Jul 22 16:07:17.652846 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.652039 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 16:07:17.653570 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.653097 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.653570 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.653107 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 16:07:17.653570 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.653333 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mc5d4\"" Jul 22 16:07:17.653570 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.653532 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 16:07:17.653836 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.653742 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:07:17.655061 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.655042 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:17.655146 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.655129 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:17.656933 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.656907 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:17.657008 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.656966 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:17.662011 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.659018 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 16:07:17.662011 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.659903 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9xbmt" Jul 22 16:07:17.662011 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.660074 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 16:07:17.662011 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.660190 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sfcqh\"" Jul 22 16:07:17.663271 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.663145 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:17.663271 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.663235 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:17.668668 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.668035 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.670300 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.670275 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:07:17.670531 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.670516 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j8z9t" Jul 22 16:07:17.673300 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.673277 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 16:07:17.673392 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.673314 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 16:07:17.673757 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.673454 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 16:07:17.673757 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.673495 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:07:17.673757 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.673628 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 16:07:17.673757 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.673673 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 16:07:17.673757 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.673736 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6m6lz\"" Jul 22 16:07:17.674028 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.673822 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 16:07:17.674028 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.673923 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 16:07:17.674119 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.674027 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v5rm7\"" Jul 22 16:07:17.674119 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.674057 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 16:07:17.674215 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.674157 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 16:07:17.674215 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.674172 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 16:07:17.675169 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.674764 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xspks" Jul 22 16:07:17.678028 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.678001 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n9fbq\"" Jul 22 16:07:17.686972 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.686568 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 16:07:17.686972 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.686815 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 16:07:17.686972 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.686875 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 16:07:17.687178 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.687070 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-g8hlf\"" Jul 22 16:07:17.715126 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.715081 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 16:07:17.723932 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.723911 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-run\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.724037 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.723949 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-sys\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.724037 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.723973 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-slash\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.724037 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6052e9bc-e447-449b-86eb-d4484c381c9c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.724037 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724028 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a19af359-cd18-4594-817d-a115665176bd-dbus\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:17.724227 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724049 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-run-k8s-cni-cncf-io\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.724227 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724071 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec-iptables-alerter-script\") pod \"iptables-alerter-9xbmt\" (UID: \"cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec\") " pod="openshift-network-operator/iptables-alerter-9xbmt" Jul 22 16:07:17.724227 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724093 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec-host-slash\") pod \"iptables-alerter-9xbmt\" (UID: \"cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec\") " pod="openshift-network-operator/iptables-alerter-9xbmt" Jul 22 16:07:17.724227 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724130 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-var-lib-cni-bin\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.724227 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724154 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqvkg\" (UniqueName: \"kubernetes.io/projected/df7d36f9-eea8-4389-9631-d91bd24acc47-kube-api-access-bqvkg\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.724227 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724190 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-sysconfig\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.724227 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724212 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-sysctl-d\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.724538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724240 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6052e9bc-e447-449b-86eb-d4484c381c9c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.724538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724285 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808dd442-3437-4347-a145-3cf2f7315f6c-tmp-dir\") pod \"node-resolver-j8z9t\" (UID: \"808dd442-3437-4347-a145-3cf2f7315f6c\") " pod="openshift-dns/node-resolver-j8z9t" Jul 22 16:07:17.724538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724309 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.724538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724331 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33854df6-9972-4671-9749-2039970fd80c-ovnkube-config\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.724538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724353 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-hostroot\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.724538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724424 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhlrh\" (UniqueName: \"kubernetes.io/projected/d0d50a09-e266-4478-8e93-19b3aea03d79-kube-api-access-zhlrh\") pod \"node-ca-xspks\" (UID: \"d0d50a09-e266-4478-8e93-19b3aea03d79\") " pod="openshift-image-registry/node-ca-xspks" Jul 22 16:07:17.724538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724447 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-var-lib-kubelet\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.724538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724488 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs\") pod \"network-metrics-daemon-6vr2t\" (UID: \"8c507486-5095-4e26-ba26-9ac77c17590d\") " pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:17.724538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724509 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-etc-kubernetes\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.724538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724531 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-device-dir\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.725051 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724552 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d0d50a09-e266-4478-8e93-19b3aea03d79-serviceca\") pod \"node-ca-xspks\" (UID: \"d0d50a09-e266-4478-8e93-19b3aea03d79\") " pod="openshift-image-registry/node-ca-xspks" Jul 22 16:07:17.725051 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724575 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6052e9bc-e447-449b-86eb-d4484c381c9c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.725051 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724621 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs46p\" (UniqueName: \"kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p\") pod \"network-check-target-rnnbk\" (UID: \"fb688b7c-f404-456b-a23e-933a5c2c7fae\") " pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:17.725051 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724652 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-etc-openvswitch\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.725051 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724684 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33854df6-9972-4671-9749-2039970fd80c-ovn-node-metrics-cert\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.725051 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724742 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89zh\" (UniqueName: \"kubernetes.io/projected/33854df6-9972-4671-9749-2039970fd80c-kube-api-access-n89zh\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.725051 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724789 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-multus-cni-dir\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.725051 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724830 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a19af359-cd18-4594-817d-a115665176bd-kubelet-config\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:17.725051 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724867 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-systemd-units\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.725051 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724898 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33854df6-9972-4671-9749-2039970fd80c-ovnkube-script-lib\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.725051 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724943 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/202823cb-4a32-4497-b506-8f0e4ed12987-etc-tuned\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.725051 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.724973 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-run-systemd\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.725575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725119 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/808dd442-3437-4347-a145-3cf2f7315f6c-hosts-file\") pod \"node-resolver-j8z9t\" (UID: \"808dd442-3437-4347-a145-3cf2f7315f6c\") " pod="openshift-dns/node-resolver-j8z9t" Jul 22 16:07:17.725575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725169 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cmkm\" (UniqueName: \"kubernetes.io/projected/808dd442-3437-4347-a145-3cf2f7315f6c-kube-api-access-2cmkm\") pod \"node-resolver-j8z9t\" (UID: \"808dd442-3437-4347-a145-3cf2f7315f6c\") " pod="openshift-dns/node-resolver-j8z9t" Jul 22 16:07:17.725575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725231 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-system-cni-dir\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.725575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725264 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-socket-dir\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.725575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725290 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-kubernetes\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.725575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725332 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxs7c\" (UniqueName: \"kubernetes.io/projected/202823cb-4a32-4497-b506-8f0e4ed12987-kube-api-access-qxs7c\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.725575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725369 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6052e9bc-e447-449b-86eb-d4484c381c9c-cnibin\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.725575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725405 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-run-multus-certs\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.725575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725443 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-systemd\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.725575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725496 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/202823cb-4a32-4497-b506-8f0e4ed12987-tmp\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.725575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725535 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6052e9bc-e447-449b-86eb-d4484c381c9c-system-cni-dir\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.725575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725567 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725600 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-run-netns\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725636 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-multus-conf-dir\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725725 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-etc-selinux\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725750 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-sys-fs\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725774 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/278b6285-2b8e-4fc3-a861-2f6088d38721-konnectivity-ca\") pod \"konnectivity-agent-vbpf9\" (UID: \"278b6285-2b8e-4fc3-a861-2f6088d38721\") " pod="kube-system/konnectivity-agent-vbpf9" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725848 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33854df6-9972-4671-9749-2039970fd80c-env-overrides\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725874 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-os-release\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725912 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-var-lib-kubelet\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725937 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df7d36f9-eea8-4389-9631-d91bd24acc47-multus-daemon-config\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.725967 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-run-ovn-kubernetes\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726021 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-cni-netd\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726054 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbv9d\" (UniqueName: \"kubernetes.io/projected/cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec-kube-api-access-bbv9d\") pod \"iptables-alerter-9xbmt\" (UID: \"cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec\") " pod="openshift-network-operator/iptables-alerter-9xbmt" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726077 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0d50a09-e266-4478-8e93-19b3aea03d79-host\") pod \"node-ca-xspks\" (UID: \"d0d50a09-e266-4478-8e93-19b3aea03d79\") " pod="openshift-image-registry/node-ca-xspks" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726108 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-modprobe-d\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.726128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726131 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-log-socket\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726154 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6052e9bc-e447-449b-86eb-d4484c381c9c-os-release\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726193 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6052e9bc-e447-449b-86eb-d4484c381c9c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726220 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcwdn\" (UniqueName: \"kubernetes.io/projected/8c507486-5095-4e26-ba26-9ac77c17590d-kube-api-access-xcwdn\") pod \"network-metrics-daemon-6vr2t\" (UID: \"8c507486-5095-4e26-ba26-9ac77c17590d\") " pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726242 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-cni-bin\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726275 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mnvk\" (UniqueName: \"kubernetes.io/projected/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-kube-api-access-2mnvk\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726419 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-sysctl-conf\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726448 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-lib-modules\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726470 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-kubelet\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726499 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-var-lib-openvswitch\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726525 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-node-log\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726561 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-cnibin\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726584 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-multus-socket-dir-parent\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726607 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-registration-dir\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726640 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/278b6285-2b8e-4fc3-a861-2f6088d38721-agent-certs\") pod \"konnectivity-agent-vbpf9\" (UID: \"278b6285-2b8e-4fc3-a861-2f6088d38721\") " pod="kube-system/konnectivity-agent-vbpf9" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726666 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-var-lib-cni-multus\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.726868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726712 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-run-openvswitch\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.727593 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726744 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.727593 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726778 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-host\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.727593 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726807 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddgpn\" (UniqueName: \"kubernetes.io/projected/6052e9bc-e447-449b-86eb-d4484c381c9c-kube-api-access-ddgpn\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.727593 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726828 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-run-ovn\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.727593 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726858 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df7d36f9-eea8-4389-9631-d91bd24acc47-cni-binary-copy\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.727593 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.726879 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-run-netns\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.814351 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.814322 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 16:07:17.827895 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.827861 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddgpn\" (UniqueName: \"kubernetes.io/projected/6052e9bc-e447-449b-86eb-d4484c381c9c-kube-api-access-ddgpn\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.828067 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.827904 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-run-ovn\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.828067 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.827932 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df7d36f9-eea8-4389-9631-d91bd24acc47-cni-binary-copy\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.828067 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.827957 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-run-netns\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.828067 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.827983 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-run\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.828067 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828005 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-sys\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.828067 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828028 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-slash\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.828067 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828050 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-run-ovn\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.828410 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828108 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-run-netns\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.828410 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828242 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-sys\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.828410 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828255 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-run\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.828410 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828298 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-slash\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.828410 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828053 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6052e9bc-e447-449b-86eb-d4484c381c9c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.828410 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828332 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a19af359-cd18-4594-817d-a115665176bd-dbus\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:17.828410 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828356 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-run-k8s-cni-cncf-io\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.828410 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828385 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec-iptables-alerter-script\") pod \"iptables-alerter-9xbmt\" (UID: \"cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec\") " pod="openshift-network-operator/iptables-alerter-9xbmt" Jul 22 16:07:17.828410 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828408 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec-host-slash\") pod \"iptables-alerter-9xbmt\" (UID: \"cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec\") " pod="openshift-network-operator/iptables-alerter-9xbmt" Jul 22 16:07:17.828898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828431 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-var-lib-cni-bin\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.828898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828453 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqvkg\" (UniqueName: \"kubernetes.io/projected/df7d36f9-eea8-4389-9631-d91bd24acc47-kube-api-access-bqvkg\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.828898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828475 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-sysconfig\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.828898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828496 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-sysctl-d\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.828898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828517 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6052e9bc-e447-449b-86eb-d4484c381c9c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.828898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828541 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808dd442-3437-4347-a145-3cf2f7315f6c-tmp-dir\") pod \"node-resolver-j8z9t\" (UID: \"808dd442-3437-4347-a145-3cf2f7315f6c\") " pod="openshift-dns/node-resolver-j8z9t" Jul 22 16:07:17.828898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828563 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.828898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828598 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33854df6-9972-4671-9749-2039970fd80c-ovnkube-config\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.828898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828622 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-hostroot\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.828898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828650 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhlrh\" (UniqueName: \"kubernetes.io/projected/d0d50a09-e266-4478-8e93-19b3aea03d79-kube-api-access-zhlrh\") pod \"node-ca-xspks\" (UID: \"d0d50a09-e266-4478-8e93-19b3aea03d79\") " pod="openshift-image-registry/node-ca-xspks" Jul 22 16:07:17.828898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828659 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6052e9bc-e447-449b-86eb-d4484c381c9c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.828898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828824 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-var-lib-kubelet\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.828898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828903 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-var-lib-kubelet\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.829466 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828906 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df7d36f9-eea8-4389-9631-d91bd24acc47-cni-binary-copy\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.829466 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.828943 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-sysconfig\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.829466 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829073 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6052e9bc-e447-449b-86eb-d4484c381c9c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.829466 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829126 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-run-k8s-cni-cncf-io\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.829466 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829160 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808dd442-3437-4347-a145-3cf2f7315f6c-tmp-dir\") pod \"node-resolver-j8z9t\" (UID: \"808dd442-3437-4347-a145-3cf2f7315f6c\") " pod="openshift-dns/node-resolver-j8z9t" Jul 22 16:07:17.829466 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829166 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec-host-slash\") pod \"iptables-alerter-9xbmt\" (UID: \"cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec\") " pod="openshift-network-operator/iptables-alerter-9xbmt" Jul 22 16:07:17.829466 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829184 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-sysctl-d\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.829466 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829379 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-var-lib-cni-bin\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.829843 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829509 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a19af359-cd18-4594-817d-a115665176bd-dbus\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:17.829843 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829548 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-hostroot\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.829843 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829584 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.829843 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829620 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs\") pod \"network-metrics-daemon-6vr2t\" (UID: \"8c507486-5095-4e26-ba26-9ac77c17590d\") " pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:17.829843 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829632 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec-iptables-alerter-script\") pod \"iptables-alerter-9xbmt\" (UID: \"cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec\") " pod="openshift-network-operator/iptables-alerter-9xbmt" Jul 22 16:07:17.829843 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829660 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-etc-kubernetes\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.829843 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.829717 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:07:17.829843 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829728 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33854df6-9972-4671-9749-2039970fd80c-ovnkube-config\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.829843 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829759 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-device-dir\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.829843 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.829784 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs podName:8c507486-5095-4e26-ba26-9ac77c17590d nodeName:}" failed. No retries permitted until 2026-07-22 16:07:18.329766157 +0000 UTC m=+3.471640212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs") pod "network-metrics-daemon-6vr2t" (UID: "8c507486-5095-4e26-ba26-9ac77c17590d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:07:17.829843 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829784 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-etc-kubernetes\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.829843 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829815 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d0d50a09-e266-4478-8e93-19b3aea03d79-serviceca\") pod \"node-ca-xspks\" (UID: \"d0d50a09-e266-4478-8e93-19b3aea03d79\") " pod="openshift-image-registry/node-ca-xspks" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829866 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6052e9bc-e447-449b-86eb-d4484c381c9c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829871 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-device-dir\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829905 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs46p\" (UniqueName: \"kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p\") pod \"network-check-target-rnnbk\" (UID: \"fb688b7c-f404-456b-a23e-933a5c2c7fae\") " pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.829995 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-etc-openvswitch\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830020 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33854df6-9972-4671-9749-2039970fd80c-ovn-node-metrics-cert\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830042 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89zh\" (UniqueName: \"kubernetes.io/projected/33854df6-9972-4671-9749-2039970fd80c-kube-api-access-n89zh\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830064 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-multus-cni-dir\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830085 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a19af359-cd18-4594-817d-a115665176bd-kubelet-config\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830100 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-systemd-units\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830117 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33854df6-9972-4671-9749-2039970fd80c-ovnkube-script-lib\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830135 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/202823cb-4a32-4497-b506-8f0e4ed12987-etc-tuned\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830150 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-run-systemd\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830169 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/808dd442-3437-4347-a145-3cf2f7315f6c-hosts-file\") pod \"node-resolver-j8z9t\" (UID: \"808dd442-3437-4347-a145-3cf2f7315f6c\") " pod="openshift-dns/node-resolver-j8z9t" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830188 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cmkm\" (UniqueName: \"kubernetes.io/projected/808dd442-3437-4347-a145-3cf2f7315f6c-kube-api-access-2cmkm\") pod \"node-resolver-j8z9t\" (UID: \"808dd442-3437-4347-a145-3cf2f7315f6c\") " pod="openshift-dns/node-resolver-j8z9t" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830210 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-system-cni-dir\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830217 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a19af359-cd18-4594-817d-a115665176bd-kubelet-config\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:17.830424 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830235 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-socket-dir\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830259 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-kubernetes\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830267 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-etc-openvswitch\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830273 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxs7c\" (UniqueName: \"kubernetes.io/projected/202823cb-4a32-4497-b506-8f0e4ed12987-kube-api-access-qxs7c\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830266 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d0d50a09-e266-4478-8e93-19b3aea03d79-serviceca\") pod \"node-ca-xspks\" (UID: \"d0d50a09-e266-4478-8e93-19b3aea03d79\") " pod="openshift-image-registry/node-ca-xspks" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830319 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6052e9bc-e447-449b-86eb-d4484c381c9c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830326 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/808dd442-3437-4347-a145-3cf2f7315f6c-hosts-file\") pod \"node-resolver-j8z9t\" (UID: \"808dd442-3437-4347-a145-3cf2f7315f6c\") " pod="openshift-dns/node-resolver-j8z9t" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830358 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-systemd-units\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830436 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-socket-dir\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830509 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-multus-cni-dir\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830502 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-kubernetes\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830614 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-run-systemd\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830810 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830923 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33854df6-9972-4671-9749-2039970fd80c-ovnkube-script-lib\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830928 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-system-cni-dir\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830967 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6052e9bc-e447-449b-86eb-d4484c381c9c-cnibin\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.830983 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6052e9bc-e447-449b-86eb-d4484c381c9c-cnibin\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831026 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-run-multus-certs\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.831209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831051 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-systemd\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831073 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/202823cb-4a32-4497-b506-8f0e4ed12987-tmp\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831105 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6052e9bc-e447-449b-86eb-d4484c381c9c-system-cni-dir\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831109 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-run-multus-certs\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831121 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-systemd\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831162 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6052e9bc-e447-449b-86eb-d4484c381c9c-system-cni-dir\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831125 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.831188 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831193 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-run-netns\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831215 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-multus-conf-dir\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831239 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-etc-selinux\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831267 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-sys-fs\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831293 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/278b6285-2b8e-4fc3-a861-2f6088d38721-konnectivity-ca\") pod \"konnectivity-agent-vbpf9\" (UID: \"278b6285-2b8e-4fc3-a861-2f6088d38721\") " pod="kube-system/konnectivity-agent-vbpf9" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.831317 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret podName:a19af359-cd18-4594-817d-a115665176bd nodeName:}" failed. No retries permitted until 2026-07-22 16:07:18.331295794 +0000 UTC m=+3.473169858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret") pod "global-pull-secret-syncer-f7rz8" (UID: "a19af359-cd18-4594-817d-a115665176bd") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831349 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33854df6-9972-4671-9749-2039970fd80c-env-overrides\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831373 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-os-release\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831396 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-var-lib-kubelet\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.832030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831419 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df7d36f9-eea8-4389-9631-d91bd24acc47-multus-daemon-config\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831444 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-run-ovn-kubernetes\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831467 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-cni-netd\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831490 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbv9d\" (UniqueName: \"kubernetes.io/projected/cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec-kube-api-access-bbv9d\") pod \"iptables-alerter-9xbmt\" (UID: \"cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec\") " pod="openshift-network-operator/iptables-alerter-9xbmt" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831511 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0d50a09-e266-4478-8e93-19b3aea03d79-host\") pod \"node-ca-xspks\" (UID: \"d0d50a09-e266-4478-8e93-19b3aea03d79\") " pod="openshift-image-registry/node-ca-xspks" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831531 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-modprobe-d\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831570 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-log-socket\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831592 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6052e9bc-e447-449b-86eb-d4484c381c9c-os-release\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831615 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6052e9bc-e447-449b-86eb-d4484c381c9c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831640 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcwdn\" (UniqueName: \"kubernetes.io/projected/8c507486-5095-4e26-ba26-9ac77c17590d-kube-api-access-xcwdn\") pod \"network-metrics-daemon-6vr2t\" (UID: \"8c507486-5095-4e26-ba26-9ac77c17590d\") " pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831661 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-cni-bin\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831683 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mnvk\" (UniqueName: \"kubernetes.io/projected/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-kube-api-access-2mnvk\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831721 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-sysctl-conf\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831742 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/278b6285-2b8e-4fc3-a861-2f6088d38721-konnectivity-ca\") pod \"konnectivity-agent-vbpf9\" (UID: \"278b6285-2b8e-4fc3-a861-2f6088d38721\") " pod="kube-system/konnectivity-agent-vbpf9" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831744 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-lib-modules\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831790 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-kubelet\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831810 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-var-lib-openvswitch\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.832784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831832 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-node-log\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831843 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-lib-modules\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831875 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-cnibin\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831879 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-node-log\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831898 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-multus-socket-dir-parent\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831912 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-run-netns\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831923 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-registration-dir\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831946 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/278b6285-2b8e-4fc3-a861-2f6088d38721-agent-certs\") pod \"konnectivity-agent-vbpf9\" (UID: \"278b6285-2b8e-4fc3-a861-2f6088d38721\") " pod="kube-system/konnectivity-agent-vbpf9" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831970 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-var-lib-cni-multus\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831992 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-run-openvswitch\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832018 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-etc-selinux\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832016 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832058 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-host\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832064 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-sys-fs\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832096 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-kubelet\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832117 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-var-lib-openvswitch\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832139 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-host\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.833385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832159 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-multus-socket-dir-parent\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832183 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-registration-dir\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.831947 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-multus-conf-dir\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832243 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-run-ovn-kubernetes\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832367 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-cni-bin\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832587 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6052e9bc-e447-449b-86eb-d4484c381c9c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832627 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33854df6-9972-4671-9749-2039970fd80c-env-overrides\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832643 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-sysctl-conf\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832650 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-cni-netd\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832710 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0d50a09-e266-4478-8e93-19b3aea03d79-host\") pod \"node-ca-xspks\" (UID: \"d0d50a09-e266-4478-8e93-19b3aea03d79\") " pod="openshift-image-registry/node-ca-xspks" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832712 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-var-lib-kubelet\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832727 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-cnibin\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832803 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/202823cb-4a32-4497-b506-8f0e4ed12987-etc-modprobe-d\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832841 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-log-socket\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832889 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6052e9bc-e447-449b-86eb-d4484c381c9c-os-release\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832929 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-run-openvswitch\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832963 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-host-var-lib-cni-multus\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832982 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df7d36f9-eea8-4389-9631-d91bd24acc47-os-release\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.834135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.832999 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33854df6-9972-4671-9749-2039970fd80c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.834988 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.833196 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df7d36f9-eea8-4389-9631-d91bd24acc47-multus-daemon-config\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.835040 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.834985 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33854df6-9972-4671-9749-2039970fd80c-ovn-node-metrics-cert\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.835192 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.835152 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/202823cb-4a32-4497-b506-8f0e4ed12987-tmp\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.835436 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.835362 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/278b6285-2b8e-4fc3-a861-2f6088d38721-agent-certs\") pod \"konnectivity-agent-vbpf9\" (UID: \"278b6285-2b8e-4fc3-a861-2f6088d38721\") " pod="kube-system/konnectivity-agent-vbpf9" Jul 22 16:07:17.836056 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.836027 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/202823cb-4a32-4497-b506-8f0e4ed12987-etc-tuned\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.840502 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.840476 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddgpn\" (UniqueName: \"kubernetes.io/projected/6052e9bc-e447-449b-86eb-d4484c381c9c-kube-api-access-ddgpn\") pod \"multus-additional-cni-plugins-xq6wn\" (UID: \"6052e9bc-e447-449b-86eb-d4484c381c9c\") " pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:17.840627 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.840561 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqvkg\" (UniqueName: \"kubernetes.io/projected/df7d36f9-eea8-4389-9631-d91bd24acc47-kube-api-access-bqvkg\") pod \"multus-pr97d\" (UID: \"df7d36f9-eea8-4389-9631-d91bd24acc47\") " pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.843184 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.843158 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 16:07:17.844904 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.844875 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhlrh\" (UniqueName: \"kubernetes.io/projected/d0d50a09-e266-4478-8e93-19b3aea03d79-kube-api-access-zhlrh\") pod \"node-ca-xspks\" (UID: \"d0d50a09-e266-4478-8e93-19b3aea03d79\") " pod="openshift-image-registry/node-ca-xspks" Jul 22 16:07:17.845180 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.845117 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:07:17.845180 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.845140 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:07:17.845180 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.845154 2567 projected.go:196] Error preparing data for projected volume kube-api-access-xs46p for pod openshift-network-diagnostics/network-check-target-rnnbk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:07:17.845355 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:17.845231 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p podName:fb688b7c-f404-456b-a23e-933a5c2c7fae nodeName:}" failed. No retries permitted until 2026-07-22 16:07:18.345209532 +0000 UTC m=+3.487083603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xs46p" (UniqueName: "kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p") pod "network-check-target-rnnbk" (UID: "fb688b7c-f404-456b-a23e-933a5c2c7fae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:07:17.846030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.846007 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cmkm\" (UniqueName: \"kubernetes.io/projected/808dd442-3437-4347-a145-3cf2f7315f6c-kube-api-access-2cmkm\") pod \"node-resolver-j8z9t\" (UID: \"808dd442-3437-4347-a145-3cf2f7315f6c\") " pod="openshift-dns/node-resolver-j8z9t" Jul 22 16:07:17.846474 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.846452 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxs7c\" (UniqueName: \"kubernetes.io/projected/202823cb-4a32-4497-b506-8f0e4ed12987-kube-api-access-qxs7c\") pod \"tuned-vjhsc\" (UID: \"202823cb-4a32-4497-b506-8f0e4ed12987\") " pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.847075 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.847054 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89zh\" (UniqueName: \"kubernetes.io/projected/33854df6-9972-4671-9749-2039970fd80c-kube-api-access-n89zh\") pod \"ovnkube-node-g2pjb\" (UID: \"33854df6-9972-4671-9749-2039970fd80c\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:17.851070 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.851044 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbv9d\" (UniqueName: \"kubernetes.io/projected/cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec-kube-api-access-bbv9d\") pod \"iptables-alerter-9xbmt\" (UID: \"cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec\") " pod="openshift-network-operator/iptables-alerter-9xbmt" Jul 22 16:07:17.852745 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.852717 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mnvk\" (UniqueName: \"kubernetes.io/projected/98bc3e49-f0e2-46de-aebe-b1d0ec810ad9-kube-api-access-2mnvk\") pod \"aws-ebs-csi-driver-node-54pxv\" (UID: \"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.853843 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.853820 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcwdn\" (UniqueName: \"kubernetes.io/projected/8c507486-5095-4e26-ba26-9ac77c17590d-kube-api-access-xcwdn\") pod \"network-metrics-daemon-6vr2t\" (UID: \"8c507486-5095-4e26-ba26-9ac77c17590d\") " pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:17.880222 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.880190 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hr82" Jul 22 16:07:17.892721 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.892674 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hr82" Jul 22 16:07:17.927199 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.927165 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:07:17.957886 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.957809 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pr97d" Jul 22 16:07:17.968714 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.968672 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" Jul 22 16:07:17.976827 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.976791 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" Jul 22 16:07:17.987282 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.987255 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vbpf9" Jul 22 16:07:17.994933 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:17.994906 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xq6wn" Jul 22 16:07:18.005584 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.005554 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9xbmt" Jul 22 16:07:18.012185 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.012161 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:18.019982 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.019961 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j8z9t" Jul 22 16:07:18.023731 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.023713 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xspks" Jul 22 16:07:18.336177 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.336139 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs\") pod \"network-metrics-daemon-6vr2t\" (UID: \"8c507486-5095-4e26-ba26-9ac77c17590d\") " pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:18.336350 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.336202 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:18.336350 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:18.336310 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:07:18.336474 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:18.336375 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret podName:a19af359-cd18-4594-817d-a115665176bd nodeName:}" failed. No retries permitted until 2026-07-22 16:07:19.336357742 +0000 UTC m=+4.478231798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret") pod "global-pull-secret-syncer-f7rz8" (UID: "a19af359-cd18-4594-817d-a115665176bd") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:07:18.336474 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:18.336310 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:07:18.336474 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:18.336464 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs podName:8c507486-5095-4e26-ba26-9ac77c17590d nodeName:}" failed. No retries permitted until 2026-07-22 16:07:19.336444736 +0000 UTC m=+4.478318793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs") pod "network-metrics-daemon-6vr2t" (UID: "8c507486-5095-4e26-ba26-9ac77c17590d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:07:18.437487 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.437459 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs46p\" (UniqueName: \"kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p\") pod \"network-check-target-rnnbk\" (UID: \"fb688b7c-f404-456b-a23e-933a5c2c7fae\") " pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:18.437642 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:18.437584 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:07:18.437642 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:18.437603 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:07:18.437642 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:18.437614 2567 projected.go:196] Error preparing data for projected volume kube-api-access-xs46p for pod openshift-network-diagnostics/network-check-target-rnnbk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:07:18.437765 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:18.437665 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p podName:fb688b7c-f404-456b-a23e-933a5c2c7fae nodeName:}" failed. No retries permitted until 2026-07-22 16:07:19.437651757 +0000 UTC m=+4.579525812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xs46p" (UniqueName: "kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p") pod "network-check-target-rnnbk" (UID: "fb688b7c-f404-456b-a23e-933a5c2c7fae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:07:18.714349 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:18.714318 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7d36f9_eea8_4389_9631_d91bd24acc47.slice/crio-af1bd094d81292a92841d20548e0903d6bddcaf3da10f8f84ff0a56e991640c8 WatchSource:0}: Error finding container af1bd094d81292a92841d20548e0903d6bddcaf3da10f8f84ff0a56e991640c8: Status 404 returned error can't find the container with id af1bd094d81292a92841d20548e0903d6bddcaf3da10f8f84ff0a56e991640c8 Jul 22 16:07:18.715300 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:18.715175 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0c2cb0_42fe_40d3_99cb_99dc1d39f7ec.slice/crio-599401c95c0b68a713592fd903478d43215ce6998a90eb80619c05b946bdf3f8 WatchSource:0}: Error finding container 599401c95c0b68a713592fd903478d43215ce6998a90eb80619c05b946bdf3f8: Status 404 returned error can't find the container with id 599401c95c0b68a713592fd903478d43215ce6998a90eb80619c05b946bdf3f8 Jul 22 16:07:18.716054 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:18.715963 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6052e9bc_e447_449b_86eb_d4484c381c9c.slice/crio-10dba692c3a2bf0c7b381d51ffb94661a2cc0e135f97b2639868b1e99e916fe9 WatchSource:0}: Error finding container 10dba692c3a2bf0c7b381d51ffb94661a2cc0e135f97b2639868b1e99e916fe9: Status 404 returned error can't find the container with id 10dba692c3a2bf0c7b381d51ffb94661a2cc0e135f97b2639868b1e99e916fe9 Jul 22 16:07:18.719002 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:18.718826 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bc3e49_f0e2_46de_aebe_b1d0ec810ad9.slice/crio-e1a8ad582ce09fc30990dc6991ab4928cf56e94bac2d6cf4c8b68f3a6d500664 WatchSource:0}: Error finding container e1a8ad582ce09fc30990dc6991ab4928cf56e94bac2d6cf4c8b68f3a6d500664: Status 404 returned error can't find the container with id e1a8ad582ce09fc30990dc6991ab4928cf56e94bac2d6cf4c8b68f3a6d500664 Jul 22 16:07:18.720425 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:18.720288 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278b6285_2b8e_4fc3_a861_2f6088d38721.slice/crio-17f18b89803cc5762f17b638cbc904f1fc47d739be641fb60fdac088e4d4929f WatchSource:0}: Error finding container 17f18b89803cc5762f17b638cbc904f1fc47d739be641fb60fdac088e4d4929f: Status 404 returned error can't find the container with id 17f18b89803cc5762f17b638cbc904f1fc47d739be641fb60fdac088e4d4929f Jul 22 16:07:18.721538 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:18.721514 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d50a09_e266_4478_8e93_19b3aea03d79.slice/crio-eb3f8131f92fb2db35c77bc89249d128cf20bde1f3a2328682e382cd08a59bca WatchSource:0}: Error finding container eb3f8131f92fb2db35c77bc89249d128cf20bde1f3a2328682e382cd08a59bca: Status 404 returned error can't find the container with id eb3f8131f92fb2db35c77bc89249d128cf20bde1f3a2328682e382cd08a59bca Jul 22 16:07:18.721857 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:18.721840 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202823cb_4a32_4497_b506_8f0e4ed12987.slice/crio-ea4d67b6c5762a576e48a8b477b6ec0e7562ac99f5bfbe874f176f38911edd29 WatchSource:0}: Error finding container ea4d67b6c5762a576e48a8b477b6ec0e7562ac99f5bfbe874f176f38911edd29: Status 404 returned error can't find the container with id ea4d67b6c5762a576e48a8b477b6ec0e7562ac99f5bfbe874f176f38911edd29 Jul 22 16:07:18.743479 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:18.743444 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808dd442_3437_4347_a145_3cf2f7315f6c.slice/crio-1511de085cb65936ba84dc63a2a9cb3da3f4119b23de4dac660245a9981cdd6f WatchSource:0}: Error finding container 1511de085cb65936ba84dc63a2a9cb3da3f4119b23de4dac660245a9981cdd6f: Status 404 returned error can't find the container with id 1511de085cb65936ba84dc63a2a9cb3da3f4119b23de4dac660245a9981cdd6f Jul 22 16:07:18.744671 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:18.744566 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33854df6_9972_4671_9749_2039970fd80c.slice/crio-89b2e8209ed0b2b039953b70c6a12af617265c9e08ed61a3faa023ab7908f637 WatchSource:0}: Error finding container 89b2e8209ed0b2b039953b70c6a12af617265c9e08ed61a3faa023ab7908f637: Status 404 returned error can't find the container with id 89b2e8209ed0b2b039953b70c6a12af617265c9e08ed61a3faa023ab7908f637 Jul 22 16:07:18.898884 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.896668 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 16:02:17 +0000 UTC" deadline="2027-12-25 09:45:07.032728024 +0000 UTC" Jul 22 16:07:18.898884 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.896933 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12497h37m48.1358s" Jul 22 16:07:18.977911 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.977822 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:18.978039 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:18.977931 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:18.986096 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.986064 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" event={"ID":"202823cb-4a32-4497-b506-8f0e4ed12987","Type":"ContainerStarted","Data":"ea4d67b6c5762a576e48a8b477b6ec0e7562ac99f5bfbe874f176f38911edd29"} Jul 22 16:07:18.987047 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.987026 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq6wn" event={"ID":"6052e9bc-e447-449b-86eb-d4484c381c9c","Type":"ContainerStarted","Data":"10dba692c3a2bf0c7b381d51ffb94661a2cc0e135f97b2639868b1e99e916fe9"} Jul 22 16:07:18.988834 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.988814 2567 generic.go:356] "Generic (PLEG): container finished" podID="59f68753e88fdefdb4f8aaf388e35bab" containerID="7cf1b92377e41a94f115d1b29136628df09340bce41dd9942a67b319494345ab" exitCode=0 Jul 22 16:07:18.988913 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.988875 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" event={"ID":"59f68753e88fdefdb4f8aaf388e35bab","Type":"ContainerDied","Data":"7cf1b92377e41a94f115d1b29136628df09340bce41dd9942a67b319494345ab"} Jul 22 16:07:18.990105 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.990015 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" event={"ID":"33854df6-9972-4671-9749-2039970fd80c","Type":"ContainerStarted","Data":"89b2e8209ed0b2b039953b70c6a12af617265c9e08ed61a3faa023ab7908f637"} Jul 22 16:07:18.991165 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.991144 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j8z9t" event={"ID":"808dd442-3437-4347-a145-3cf2f7315f6c","Type":"ContainerStarted","Data":"1511de085cb65936ba84dc63a2a9cb3da3f4119b23de4dac660245a9981cdd6f"} Jul 22 16:07:18.992269 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.992247 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xspks" event={"ID":"d0d50a09-e266-4478-8e93-19b3aea03d79","Type":"ContainerStarted","Data":"eb3f8131f92fb2db35c77bc89249d128cf20bde1f3a2328682e382cd08a59bca"} Jul 22 16:07:18.993282 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.993263 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vbpf9" event={"ID":"278b6285-2b8e-4fc3-a861-2f6088d38721","Type":"ContainerStarted","Data":"17f18b89803cc5762f17b638cbc904f1fc47d739be641fb60fdac088e4d4929f"} Jul 22 16:07:18.994340 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.994317 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" event={"ID":"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9","Type":"ContainerStarted","Data":"e1a8ad582ce09fc30990dc6991ab4928cf56e94bac2d6cf4c8b68f3a6d500664"} Jul 22 16:07:18.995332 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.995314 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9xbmt" event={"ID":"cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec","Type":"ContainerStarted","Data":"599401c95c0b68a713592fd903478d43215ce6998a90eb80619c05b946bdf3f8"} Jul 22 16:07:18.996383 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.996364 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pr97d" event={"ID":"df7d36f9-eea8-4389-9631-d91bd24acc47","Type":"ContainerStarted","Data":"af1bd094d81292a92841d20548e0903d6bddcaf3da10f8f84ff0a56e991640c8"} Jul 22 16:07:18.997611 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:18.997593 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" event={"ID":"f338029658e35a32ff779b9f1d620fb0","Type":"ContainerStarted","Data":"3e6b009e6a5739a05771a993069536fe5c51740a79bc5e585cb53c3bff234591"} Jul 22 16:07:19.344235 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:19.344193 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:19.344412 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:19.344283 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs\") pod \"network-metrics-daemon-6vr2t\" (UID: \"8c507486-5095-4e26-ba26-9ac77c17590d\") " pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:19.344472 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:19.344416 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:07:19.344520 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:19.344477 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs podName:8c507486-5095-4e26-ba26-9ac77c17590d nodeName:}" failed. No retries permitted until 2026-07-22 16:07:21.344458898 +0000 UTC m=+6.486332968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs") pod "network-metrics-daemon-6vr2t" (UID: "8c507486-5095-4e26-ba26-9ac77c17590d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:07:19.344914 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:19.344893 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:07:19.345012 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:19.344950 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret podName:a19af359-cd18-4594-817d-a115665176bd nodeName:}" failed. No retries permitted until 2026-07-22 16:07:21.344934429 +0000 UTC m=+6.486808501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret") pod "global-pull-secret-syncer-f7rz8" (UID: "a19af359-cd18-4594-817d-a115665176bd") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:07:19.445056 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:19.445017 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs46p\" (UniqueName: \"kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p\") pod \"network-check-target-rnnbk\" (UID: \"fb688b7c-f404-456b-a23e-933a5c2c7fae\") " pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:19.445262 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:19.445245 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:07:19.445325 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:19.445274 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:07:19.445325 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:19.445287 2567 projected.go:196] Error preparing data for projected volume kube-api-access-xs46p for pod openshift-network-diagnostics/network-check-target-rnnbk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:07:19.445426 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:19.445343 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p podName:fb688b7c-f404-456b-a23e-933a5c2c7fae nodeName:}" failed. No retries permitted until 2026-07-22 16:07:21.445324632 +0000 UTC m=+6.587198688 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xs46p" (UniqueName: "kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p") pod "network-check-target-rnnbk" (UID: "fb688b7c-f404-456b-a23e-933a5c2c7fae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:07:19.897990 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:19.897780 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 16:02:17 +0000 UTC" deadline="2028-02-10 09:56:25.128350893 +0000 UTC" Jul 22 16:07:19.897990 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:19.897818 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13625h49m5.230538196s" Jul 22 16:07:19.978663 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:19.978447 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:19.978663 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:19.978582 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:19.979150 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:19.979060 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:19.979228 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:19.979156 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:20.020277 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:20.020237 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" event={"ID":"59f68753e88fdefdb4f8aaf388e35bab","Type":"ContainerStarted","Data":"fef4a5c1ae90813e65d6a84731e6bcab53cdee3a68092878d61c39590e065d4c"} Jul 22 16:07:20.040747 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:20.040674 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" podStartSLOduration=3.040650086 podStartE2EDuration="3.040650086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:07:19.03131551 +0000 UTC m=+4.173189609" watchObservedRunningTime="2026-07-22 16:07:20.040650086 +0000 UTC m=+5.182524164" Jul 22 16:07:20.978838 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:20.978312 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:20.978838 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:20.978455 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:21.362351 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:21.362314 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:21.362537 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:21.362390 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs\") pod \"network-metrics-daemon-6vr2t\" (UID: \"8c507486-5095-4e26-ba26-9ac77c17590d\") " pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:21.362601 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:21.362553 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:07:21.362648 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:21.362610 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs podName:8c507486-5095-4e26-ba26-9ac77c17590d nodeName:}" failed. No retries permitted until 2026-07-22 16:07:25.362593088 +0000 UTC m=+10.504467157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs") pod "network-metrics-daemon-6vr2t" (UID: "8c507486-5095-4e26-ba26-9ac77c17590d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:07:21.362726 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:21.362674 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:07:21.363088 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:21.363072 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret podName:a19af359-cd18-4594-817d-a115665176bd nodeName:}" failed. No retries permitted until 2026-07-22 16:07:25.36305455 +0000 UTC m=+10.504928611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret") pod "global-pull-secret-syncer-f7rz8" (UID: "a19af359-cd18-4594-817d-a115665176bd") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:07:21.465026 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:21.464987 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs46p\" (UniqueName: \"kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p\") pod \"network-check-target-rnnbk\" (UID: \"fb688b7c-f404-456b-a23e-933a5c2c7fae\") " pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:21.465198 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:21.465147 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:07:21.465198 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:21.465170 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:07:21.465198 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:21.465182 2567 projected.go:196] Error preparing data for projected volume kube-api-access-xs46p for pod openshift-network-diagnostics/network-check-target-rnnbk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:07:21.465358 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:21.465244 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p podName:fb688b7c-f404-456b-a23e-933a5c2c7fae nodeName:}" failed. No retries permitted until 2026-07-22 16:07:25.465224782 +0000 UTC m=+10.607098839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xs46p" (UniqueName: "kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p") pod "network-check-target-rnnbk" (UID: "fb688b7c-f404-456b-a23e-933a5c2c7fae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:07:21.978445 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:21.977919 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:21.978445 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:21.978084 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:21.978445 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:21.978311 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:21.978445 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:21.978417 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:22.978639 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:22.978233 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:22.978639 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:22.978377 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:23.978459 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:23.978116 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:23.978459 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:23.978140 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:23.978459 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:23.978253 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:23.979085 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:23.978659 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:24.977674 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:24.977631 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:24.977879 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:24.977788 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:25.399189 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:25.399144 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:25.399655 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:25.399222 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs\") pod \"network-metrics-daemon-6vr2t\" (UID: \"8c507486-5095-4e26-ba26-9ac77c17590d\") " pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:25.399655 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:25.399348 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:07:25.399655 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:25.399437 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs podName:8c507486-5095-4e26-ba26-9ac77c17590d nodeName:}" failed. No retries permitted until 2026-07-22 16:07:33.399421936 +0000 UTC m=+18.541295991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs") pod "network-metrics-daemon-6vr2t" (UID: "8c507486-5095-4e26-ba26-9ac77c17590d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:07:25.399874 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:25.399855 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:07:25.399925 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:25.399911 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret podName:a19af359-cd18-4594-817d-a115665176bd nodeName:}" failed. No retries permitted until 2026-07-22 16:07:33.399895832 +0000 UTC m=+18.541769907 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret") pod "global-pull-secret-syncer-f7rz8" (UID: "a19af359-cd18-4594-817d-a115665176bd") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:07:25.500280 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:25.500227 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs46p\" (UniqueName: \"kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p\") pod \"network-check-target-rnnbk\" (UID: \"fb688b7c-f404-456b-a23e-933a5c2c7fae\") " pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:25.500478 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:25.500435 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:07:25.500478 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:25.500460 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:07:25.500478 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:25.500472 2567 projected.go:196] Error preparing data for projected volume kube-api-access-xs46p for pod openshift-network-diagnostics/network-check-target-rnnbk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:07:25.500585 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:25.500535 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p podName:fb688b7c-f404-456b-a23e-933a5c2c7fae nodeName:}" failed. No retries permitted until 2026-07-22 16:07:33.500518143 +0000 UTC m=+18.642392204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xs46p" (UniqueName: "kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p") pod "network-check-target-rnnbk" (UID: "fb688b7c-f404-456b-a23e-933a5c2c7fae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:07:25.979820 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:25.979195 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:25.979820 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:25.979315 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:25.979820 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:25.979669 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:25.979820 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:25.979774 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:26.977685 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:26.977599 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:26.978123 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:26.977741 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:27.977954 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:27.977919 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:27.978412 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:27.977922 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:27.978412 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:27.978036 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:27.978412 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:27.978147 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:28.978036 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:28.978001 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:28.978396 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:28.978131 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:29.977674 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:29.977638 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:29.977870 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:29.977639 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:29.977870 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:29.977782 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:29.977963 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:29.977868 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:30.977826 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:30.977789 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:30.978297 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:30.977932 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:31.978124 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:31.978087 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:31.978539 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:31.978139 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:31.978539 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:31.978228 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:31.978539 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:31.978362 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:32.977881 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:32.977854 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:32.978117 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:32.977959 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:33.465066 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:33.465029 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs\") pod \"network-metrics-daemon-6vr2t\" (UID: \"8c507486-5095-4e26-ba26-9ac77c17590d\") " pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:33.465507 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:33.465109 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:33.465507 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:33.465165 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:07:33.465507 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:33.465248 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs podName:8c507486-5095-4e26-ba26-9ac77c17590d nodeName:}" failed. No retries permitted until 2026-07-22 16:07:49.465214649 +0000 UTC m=+34.607088714 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs") pod "network-metrics-daemon-6vr2t" (UID: "8c507486-5095-4e26-ba26-9ac77c17590d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:07:33.465507 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:33.465278 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:07:33.465507 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:33.465337 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret podName:a19af359-cd18-4594-817d-a115665176bd nodeName:}" failed. No retries permitted until 2026-07-22 16:07:49.465320488 +0000 UTC m=+34.607194549 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret") pod "global-pull-secret-syncer-f7rz8" (UID: "a19af359-cd18-4594-817d-a115665176bd") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:07:33.565669 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:33.565626 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs46p\" (UniqueName: \"kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p\") pod \"network-check-target-rnnbk\" (UID: \"fb688b7c-f404-456b-a23e-933a5c2c7fae\") " pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:33.565866 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:33.565844 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:07:33.565931 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:33.565875 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:07:33.565931 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:33.565888 2567 projected.go:196] Error preparing data for projected volume kube-api-access-xs46p for pod openshift-network-diagnostics/network-check-target-rnnbk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:07:33.565994 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:33.565953 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p podName:fb688b7c-f404-456b-a23e-933a5c2c7fae nodeName:}" failed. No retries permitted until 2026-07-22 16:07:49.56593507 +0000 UTC m=+34.707809130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xs46p" (UniqueName: "kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p") pod "network-check-target-rnnbk" (UID: "fb688b7c-f404-456b-a23e-933a5c2c7fae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:07:33.977826 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:33.977784 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:33.978005 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:33.977784 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:33.978005 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:33.977907 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:33.978169 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:33.978020 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:34.978180 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:34.978146 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:34.978584 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:34.978282 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:35.978681 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:35.978641 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:35.979102 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:35.978766 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:35.979102 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:35.978946 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:35.979102 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:35.979056 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:36.978387 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:36.978352 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:36.978530 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:36.978456 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:37.980245 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:37.979983 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:37.980245 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:37.980205 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:37.981944 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:37.981221 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:37.981944 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:37.981308 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:38.058780 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.058745 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pr97d" event={"ID":"df7d36f9-eea8-4389-9631-d91bd24acc47","Type":"ContainerStarted","Data":"c77060a8a77bfc416bd6b4240ca572320f21a8f01e77f7948c67ab0fd841c41b"} Jul 22 16:07:38.060291 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.060259 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" event={"ID":"202823cb-4a32-4497-b506-8f0e4ed12987","Type":"ContainerStarted","Data":"6ac5aff578bf0384c65a531ae9c30cc4f00efc59d0776d5f89c1b595e13490be"} Jul 22 16:07:38.061868 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.061804 2567 generic.go:356] "Generic (PLEG): container finished" podID="6052e9bc-e447-449b-86eb-d4484c381c9c" containerID="a89149210a1fb6ff9fae6147fb748dfafac61e65cc70f97196eb78ab4922147b" exitCode=0 Jul 22 16:07:38.061942 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.061882 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq6wn" event={"ID":"6052e9bc-e447-449b-86eb-d4484c381c9c","Type":"ContainerDied","Data":"a89149210a1fb6ff9fae6147fb748dfafac61e65cc70f97196eb78ab4922147b"} Jul 22 16:07:38.064833 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.064816 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:07:38.065196 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.065172 2567 generic.go:356] "Generic (PLEG): container finished" podID="33854df6-9972-4671-9749-2039970fd80c" containerID="a3df13b7b5538a4f868a7ff0fb13b07b91ad930073cdf002b28abfdec63e0a73" exitCode=1 Jul 22 16:07:38.065287 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.065240 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" event={"ID":"33854df6-9972-4671-9749-2039970fd80c","Type":"ContainerStarted","Data":"985cd97aa60b42eeb4486f93e265d0a6f3b6de14704fc2ffcce6bdcca1ca1d2b"} Jul 22 16:07:38.065338 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.065301 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" event={"ID":"33854df6-9972-4671-9749-2039970fd80c","Type":"ContainerStarted","Data":"c9cd8599775ae910c4f229fb03712848bc82dea7cee88775a2242d71949bb6fe"} Jul 22 16:07:38.065395 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.065346 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" event={"ID":"33854df6-9972-4671-9749-2039970fd80c","Type":"ContainerStarted","Data":"0de972dd05813e191ccd56dc5d4b698752571e87693fa9f54fe9486ebab3b469"} Jul 22 16:07:38.065395 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.065384 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" event={"ID":"33854df6-9972-4671-9749-2039970fd80c","Type":"ContainerDied","Data":"a3df13b7b5538a4f868a7ff0fb13b07b91ad930073cdf002b28abfdec63e0a73"} Jul 22 16:07:38.065490 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.065433 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" event={"ID":"33854df6-9972-4671-9749-2039970fd80c","Type":"ContainerStarted","Data":"70603340b3715d5e6a4df3152ea1802cbb423affbbac962718d8a7f3c92cb95a"} Jul 22 16:07:38.066639 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.066617 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j8z9t" event={"ID":"808dd442-3437-4347-a145-3cf2f7315f6c","Type":"ContainerStarted","Data":"6daffa0a4b091eae39f8bf74b6283666bcaf7b053c29b01166fe218162c809b7"} Jul 22 16:07:38.068219 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.068195 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xspks" event={"ID":"d0d50a09-e266-4478-8e93-19b3aea03d79","Type":"ContainerStarted","Data":"5b5b198091e77b27673543573dbc240e433a81ec9911a8dc1ce286b00e53c6db"} Jul 22 16:07:38.069655 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.069628 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vbpf9" event={"ID":"278b6285-2b8e-4fc3-a861-2f6088d38721","Type":"ContainerStarted","Data":"0386f8a4c4665253b36d6276c4a3c51ba2403dfc40fe47a51735c8145c33eb35"} Jul 22 16:07:38.071064 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.071043 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" event={"ID":"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9","Type":"ContainerStarted","Data":"343ac51f4533778d447ad541b1fdcee537a8462e36e4cc8500109f333c52fa92"} Jul 22 16:07:38.079776 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.079733 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pr97d" podStartSLOduration=3.439864071 podStartE2EDuration="22.079721958s" totalImagesPullingTime="18.639857887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:07:38.079624028 +0000 UTC m=+23.221498105" watchObservedRunningTime="2026-07-22 16:07:38.079721958 +0000 UTC m=+23.221596034" Jul 22 16:07:38.080143 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.080115 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" podStartSLOduration=21.080106117 podStartE2EDuration="21.080106117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:07:20.041967211 +0000 UTC m=+5.183841289" watchObservedRunningTime="2026-07-22 16:07:38.080106117 +0000 UTC m=+23.221980193" Jul 22 16:07:38.095955 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.095907 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j8z9t" podStartSLOduration=3.547376778 podStartE2EDuration="22.095892225s" totalImagesPullingTime="18.548515447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:07:38.09558492 +0000 UTC m=+23.237458996" watchObservedRunningTime="2026-07-22 16:07:38.095892225 +0000 UTC m=+23.237766300" Jul 22 16:07:38.111171 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.111130 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vbpf9" podStartSLOduration=7.75200007 podStartE2EDuration="22.11111536s" totalImagesPullingTime="14.35911529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:07:38.110634183 +0000 UTC m=+23.252508266" watchObservedRunningTime="2026-07-22 16:07:38.11111536 +0000 UTC m=+23.252989437" Jul 22 16:07:38.171391 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.171341 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xspks" podStartSLOduration=3.617961359 podStartE2EDuration="22.171323888s" totalImagesPullingTime="18.553362529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:07:38.147559627 +0000 UTC m=+23.289433704" watchObservedRunningTime="2026-07-22 16:07:38.171323888 +0000 UTC m=+23.313198035" Jul 22 16:07:38.171727 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.171685 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vjhsc" podStartSLOduration=3.599454578 podStartE2EDuration="22.171676188s" totalImagesPullingTime="18.57222161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:07:38.17110714 +0000 UTC m=+23.312981216" watchObservedRunningTime="2026-07-22 16:07:38.171676188 +0000 UTC m=+23.313550265" Jul 22 16:07:38.729770 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.729745 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 16:07:38.886655 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.886447 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T16:07:38.729765413Z","UUID":"bab71c40-226a-4179-a3d9-5ea32847a5a3","Handler":null,"Name":"","Endpoint":""} Jul 22 16:07:38.888167 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.888142 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 16:07:38.888293 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.888187 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 16:07:38.978004 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:38.977962 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:38.978229 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:38.978089 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:39.075082 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:39.075050 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" event={"ID":"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9","Type":"ContainerStarted","Data":"5d1dc7725f418654bcb690c9a9b7eda3790de0c54298402fbaac05eaf628ef13"} Jul 22 16:07:39.077020 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:39.076991 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9xbmt" event={"ID":"cf0c2cb0-42fe-40d3-99cb-99dc1d39f7ec","Type":"ContainerStarted","Data":"bb84ea2c71fff07334668687682e0a430d634194b9c89aa1d8a4c50710ecb11b"} Jul 22 16:07:39.080466 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:39.080440 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:07:39.081613 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:39.081453 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" event={"ID":"33854df6-9972-4671-9749-2039970fd80c","Type":"ContainerStarted","Data":"afa2a6cd6c27ccc3b5cae1a7cabe608fbdb9f0356759f796ae87338e0d2aa8c6"} Jul 22 16:07:39.093977 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:39.093926 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9xbmt" podStartSLOduration=4.497075367 podStartE2EDuration="23.093912636s" totalImagesPullingTime="18.596837269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:07:39.093335108 +0000 UTC m=+24.235209185" watchObservedRunningTime="2026-07-22 16:07:39.093912636 +0000 UTC m=+24.235786712" Jul 22 16:07:39.238453 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:39.238328 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vbpf9" Jul 22 16:07:39.238826 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:39.238803 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vbpf9" Jul 22 16:07:39.977854 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:39.977826 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:39.977854 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:39.977842 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:39.978109 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:39.977972 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:39.978168 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:39.978115 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:40.085825 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:40.085788 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" event={"ID":"98bc3e49-f0e2-46de-aebe-b1d0ec810ad9","Type":"ContainerStarted","Data":"00ce6e742cdfcbd458f39bac19a438cf731f93508f65f8138b87d76e9f537d28"} Jul 22 16:07:40.086387 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:40.086360 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vbpf9" Jul 22 16:07:40.086822 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:40.086801 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vbpf9" Jul 22 16:07:40.108772 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:40.108720 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54pxv" podStartSLOduration=3.4258482 podStartE2EDuration="24.108704937s" totalImagesPullingTime="20.682856737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:07:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:07:40.107222443 +0000 UTC m=+25.249096540" watchObservedRunningTime="2026-07-22 16:07:40.108704937 +0000 UTC m=+25.250579007" Jul 22 16:07:40.978328 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:40.978154 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:40.978507 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:40.978417 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:41.090882 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:41.090856 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:07:41.091323 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:41.091292 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" event={"ID":"33854df6-9972-4671-9749-2039970fd80c","Type":"ContainerStarted","Data":"b654461f3a561a6da3134a62fcc5c4da63d432aaa7e1e6f760017d8782ebb78a"} Jul 22 16:07:41.977996 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:41.977963 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:41.978207 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:41.978130 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:41.978207 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:41.978175 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:41.978327 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:41.978286 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:42.978554 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:42.978357 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:42.979218 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:42.978633 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:43.100093 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:43.100066 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:07:43.100461 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:43.100430 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" event={"ID":"33854df6-9972-4671-9749-2039970fd80c","Type":"ContainerStarted","Data":"bafe632babe934f0f5a657d56968105e78bd865b345531d9b1cc0fef96433fe2"} Jul 22 16:07:43.100749 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:43.100729 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:43.100854 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:43.100824 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:43.100926 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:43.100909 2567 scope.go:117] "RemoveContainer" containerID="a3df13b7b5538a4f868a7ff0fb13b07b91ad930073cdf002b28abfdec63e0a73" Jul 22 16:07:43.102561 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:43.102537 2567 generic.go:356] "Generic (PLEG): container finished" podID="6052e9bc-e447-449b-86eb-d4484c381c9c" containerID="2f9b75ee7b8b4e2f8a40a0a6a797d3aa839611d8b799ae08993c559ac308bf7d" exitCode=0 Jul 22 16:07:43.102733 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:43.102570 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq6wn" event={"ID":"6052e9bc-e447-449b-86eb-d4484c381c9c","Type":"ContainerDied","Data":"2f9b75ee7b8b4e2f8a40a0a6a797d3aa839611d8b799ae08993c559ac308bf7d"} Jul 22 16:07:43.117353 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:43.117334 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:43.978259 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:43.978229 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:43.978504 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:43.978331 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:43.978504 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:43.978388 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:43.978577 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:43.978513 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:44.108030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:44.108003 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:07:44.108349 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:44.108321 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" event={"ID":"33854df6-9972-4671-9749-2039970fd80c","Type":"ContainerStarted","Data":"4e5b379b2c76bf421deec261432175a73ccd8f197a242a4ba032e13cc26ce800"} Jul 22 16:07:44.108561 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:44.108544 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:44.123271 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:44.123252 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:07:44.141722 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:44.141653 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" podStartSLOduration=9.401117077 podStartE2EDuration="28.141635142s" totalImagesPullingTime="18.740518065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:07:44.140830603 +0000 UTC m=+29.282704700" watchObservedRunningTime="2026-07-22 16:07:44.141635142 +0000 UTC m=+29.283509221" Jul 22 16:07:44.731320 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:44.731283 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f7rz8"] Jul 22 16:07:44.731503 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:44.731407 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:44.731566 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:44.731516 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:44.731817 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:44.731796 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rnnbk"] Jul 22 16:07:44.731900 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:44.731871 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:44.731959 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:44.731939 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:44.733075 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:44.733048 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6vr2t"] Jul 22 16:07:44.733177 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:44.733158 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:44.733286 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:44.733265 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:45.112676 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:45.112645 2567 generic.go:356] "Generic (PLEG): container finished" podID="6052e9bc-e447-449b-86eb-d4484c381c9c" containerID="183f75f2407d990ac7e11e1876be574fb17e610c2d4d0b82e2945fccd6bd981e" exitCode=0 Jul 22 16:07:45.113076 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:45.112730 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq6wn" event={"ID":"6052e9bc-e447-449b-86eb-d4484c381c9c","Type":"ContainerDied","Data":"183f75f2407d990ac7e11e1876be574fb17e610c2d4d0b82e2945fccd6bd981e"} Jul 22 16:07:45.978863 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:45.978831 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:45.979028 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:45.978904 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:45.979028 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:45.978914 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:45.979028 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:45.978994 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:45.979127 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:45.979036 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:45.979127 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:45.979092 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:47.119882 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:47.119613 2567 generic.go:356] "Generic (PLEG): container finished" podID="6052e9bc-e447-449b-86eb-d4484c381c9c" containerID="b42bac00af75c83c4405a6ad6bdaa1ee40b9ff5529406c57e54a5d5b39c78da5" exitCode=0 Jul 22 16:07:47.119882 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:47.119684 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq6wn" event={"ID":"6052e9bc-e447-449b-86eb-d4484c381c9c","Type":"ContainerDied","Data":"b42bac00af75c83c4405a6ad6bdaa1ee40b9ff5529406c57e54a5d5b39c78da5"} Jul 22 16:07:47.977659 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:47.977622 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:47.977847 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:47.977622 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:47.977847 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:47.977628 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:47.977847 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:47.977768 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:47.978014 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:47.977855 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:47.978014 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:47.977956 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:49.494021 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:49.493928 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs\") pod \"network-metrics-daemon-6vr2t\" (UID: \"8c507486-5095-4e26-ba26-9ac77c17590d\") " pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:49.494021 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:49.493994 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:49.494527 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:49.494089 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:07:49.494527 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:49.494119 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:07:49.494527 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:49.494157 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs podName:8c507486-5095-4e26-ba26-9ac77c17590d nodeName:}" failed. No retries permitted until 2026-07-22 16:08:21.494136259 +0000 UTC m=+66.636010315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs") pod "network-metrics-daemon-6vr2t" (UID: "8c507486-5095-4e26-ba26-9ac77c17590d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:07:49.494527 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:49.494173 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret podName:a19af359-cd18-4594-817d-a115665176bd nodeName:}" failed. No retries permitted until 2026-07-22 16:08:21.494166 +0000 UTC m=+66.636040055 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret") pod "global-pull-secret-syncer-f7rz8" (UID: "a19af359-cd18-4594-817d-a115665176bd") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:07:49.595347 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:49.595313 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs46p\" (UniqueName: \"kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p\") pod \"network-check-target-rnnbk\" (UID: \"fb688b7c-f404-456b-a23e-933a5c2c7fae\") " pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:49.595492 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:49.595422 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:07:49.595492 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:49.595439 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:07:49.595492 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:49.595449 2567 projected.go:196] Error preparing data for projected volume kube-api-access-xs46p for pod openshift-network-diagnostics/network-check-target-rnnbk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:07:49.595605 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:49.595498 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p podName:fb688b7c-f404-456b-a23e-933a5c2c7fae nodeName:}" failed. No retries permitted until 2026-07-22 16:08:21.595482308 +0000 UTC m=+66.737356364 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xs46p" (UniqueName: "kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p") pod "network-check-target-rnnbk" (UID: "fb688b7c-f404-456b-a23e-933a5c2c7fae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:07:49.977976 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:49.977944 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:49.978137 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:49.977944 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:49.978137 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:49.977945 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:49.978137 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:49.978069 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnbk" podUID="fb688b7c-f404-456b-a23e-933a5c2c7fae" Jul 22 16:07:49.978291 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:49.978174 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vr2t" podUID="8c507486-5095-4e26-ba26-9ac77c17590d" Jul 22 16:07:49.978291 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:49.978268 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f7rz8" podUID="a19af359-cd18-4594-817d-a115665176bd" Jul 22 16:07:51.148171 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.148102 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeReady" Jul 22 16:07:51.148587 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.148252 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 16:07:51.186486 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.186451 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b465bb546-ppslg"] Jul 22 16:07:51.190079 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.190056 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.192385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.192361 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 16:07:51.192654 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.192638 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-crmjn\"" Jul 22 16:07:51.192922 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.192906 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 16:07:51.193994 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.193973 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 16:07:51.202145 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.201439 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 16:07:51.207846 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.207791 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b465bb546-ppslg"] Jul 22 16:07:51.216071 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.216050 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c78v9"] Jul 22 16:07:51.218785 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.218761 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-586bddb798-57m2m"] Jul 22 16:07:51.218959 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.218938 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:51.221500 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.221478 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 16:07:51.222557 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.222248 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 16:07:51.222557 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.222410 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wdbjp\"" Jul 22 16:07:51.222733 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.222714 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-24cbv"] Jul 22 16:07:51.222785 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.222766 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:51.226146 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.226126 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ltpvx"] Jul 22 16:07:51.226248 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.226205 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.228440 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.228418 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 16:07:51.228517 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.228421 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 22 16:07:51.228517 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.228512 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 22 16:07:51.228666 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.228652 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 22 16:07:51.228770 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.228749 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-56t8g\"" Jul 22 16:07:51.228770 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.228766 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 22 16:07:51.228866 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.228844 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5"] Jul 22 16:07:51.228918 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.228874 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 16:07:51.229013 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.228997 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vmv54\"" Jul 22 16:07:51.229057 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.229003 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ltpvx" Jul 22 16:07:51.229404 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.229389 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 16:07:51.229448 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.229430 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 22 16:07:51.229597 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.229581 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 16:07:51.231547 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.231530 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-t5kfc\"" Jul 22 16:07:51.231639 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.231532 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:07:51.231721 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.231655 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 22 16:07:51.231857 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.231844 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq"] Jul 22 16:07:51.231983 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.231960 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" Jul 22 16:07:51.232487 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.232443 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 22 16:07:51.234427 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.234305 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 22 16:07:51.234559 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.234530 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-p6vjx\"" Jul 22 16:07:51.234652 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.234568 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 22 16:07:51.234818 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.234789 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 22 16:07:51.234909 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.234867 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:07:51.235213 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.235196 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4"] Jul 22 16:07:51.235350 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.235335 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:07:51.239130 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.238170 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889"] Jul 22 16:07:51.239130 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.238455 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" Jul 22 16:07:51.239877 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.239746 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-28lg6\"" Jul 22 16:07:51.239977 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.239887 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 16:07:51.240144 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.240126 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 16:07:51.240222 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.240184 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 22 16:07:51.241756 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.241506 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-sf2mn"] Jul 22 16:07:51.242495 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.242474 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 22 16:07:51.243592 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.243571 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 22 16:07:51.243988 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.243968 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" Jul 22 16:07:51.251719 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.245988 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 22 16:07:51.251719 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.246523 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:07:51.251719 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.248912 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c78v9"] Jul 22 16:07:51.251719 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.248960 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq"] Jul 22 16:07:51.251719 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.248988 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4"] Jul 22 16:07:51.251719 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.249018 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ltpvx"] Jul 22 16:07:51.251719 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.249053 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-9nf8d"] Jul 22 16:07:51.251719 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.249490 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 22 16:07:51.251719 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.249609 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gljqq\"" Jul 22 16:07:51.251719 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.251121 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:07:51.253398 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.253379 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vprx2"] Jul 22 16:07:51.253513 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.253498 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-9nf8d" Jul 22 16:07:51.253987 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.253966 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 22 16:07:51.254091 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.254021 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8dgvc\"" Jul 22 16:07:51.254371 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.254354 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 22 16:07:51.254428 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.254372 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 22 16:07:51.254657 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.254640 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 22 16:07:51.254927 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.254910 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:07:51.255122 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.255104 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 22 16:07:51.255224 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.255178 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2mw2r\"" Jul 22 16:07:51.255224 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.255200 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:07:51.255224 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.255184 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 22 16:07:51.255488 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.255466 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 16:07:51.255590 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.255576 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-59pzn\"" Jul 22 16:07:51.255643 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.255623 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 16:07:51.256737 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.256718 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-586bddb798-57m2m"] Jul 22 16:07:51.256808 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.256754 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-sf2mn"] Jul 22 16:07:51.256808 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.256791 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vprx2" Jul 22 16:07:51.259429 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.259411 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-24cbv"] Jul 22 16:07:51.260769 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.260336 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-9nf8d"] Jul 22 16:07:51.261223 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.261194 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889"] Jul 22 16:07:51.261934 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.261767 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 16:07:51.261934 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.261800 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 16:07:51.261934 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.261893 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 16:07:51.262385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.262367 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7zs9d\"" Jul 22 16:07:51.263240 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.263221 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 22 16:07:51.271291 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.271273 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5"] Jul 22 16:07:51.286760 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.286734 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vprx2"] Jul 22 16:07:51.307508 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.307491 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-ca-trust-extracted\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.307578 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.307516 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-installation-pull-secrets\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.307643 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.307596 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-image-registry-private-configuration\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.307643 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.307633 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv95p\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-kube-api-access-dv95p\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.307754 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.307672 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-bound-sa-token\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.307754 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.307738 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.307827 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.307759 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-certificates\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.307827 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.307785 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-trusted-ca\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.408547 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.408449 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-bound-sa-token\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.408547 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.408503 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6wk5\" (UniqueName: \"kubernetes.io/projected/dc08ab36-8b95-4b9f-a9dc-e3aade431059-kube-api-access-r6wk5\") pod \"volume-data-source-validator-78cf4c6f6-ltpvx\" (UID: \"dc08ab36-8b95-4b9f-a9dc-e3aade431059\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ltpvx" Jul 22 16:07:51.408547 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.408538 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-default-certificate\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:51.408838 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.408570 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/957c1e60-c08f-437f-afaf-d3679b2e8376-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-6s889\" (UID: \"957c1e60-c08f-437f-afaf-d3679b2e8376\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" Jul 22 16:07:51.408838 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.408601 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:51.408838 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.408628 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-stats-auth\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:51.408838 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.408664 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-serving-cert\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.408838 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.408744 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbnbm\" (UniqueName: \"kubernetes.io/projected/7e88090f-ac10-4518-954c-aa515ceb5cb7-kube-api-access-wbnbm\") pod \"cluster-samples-operator-79b7c55f87-ks2q4\" (UID: \"7e88090f-ac10-4518-954c-aa515ceb5cb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" Jul 22 16:07:51.408838 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.408796 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-trusted-ca\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.408838 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.408824 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w2gb\" (UniqueName: \"kubernetes.io/projected/57dbd4ff-f4ed-484a-801b-9cef8a961409-kube-api-access-4w2gb\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:51.409147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.408850 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/799e2b87-1954-4103-860d-33ab42170349-config\") pod \"console-operator-565b7b569d-sf2mn\" (UID: \"799e2b87-1954-4103-860d-33ab42170349\") " pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:07:51.409147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.408878 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94sgl\" (UniqueName: \"kubernetes.io/projected/0a773bcb-61a5-42df-a42c-31f53a91d776-kube-api-access-94sgl\") pod \"network-check-source-7bc5b6bf85-9nf8d\" (UID: \"0a773bcb-61a5-42df-a42c-31f53a91d776\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-9nf8d" Jul 22 16:07:51.409147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.408923 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-ca-trust-extracted\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.409147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.408991 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-config-volume\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:51.409147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409068 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rv6j\" (UniqueName: \"kubernetes.io/projected/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-kube-api-access-2rv6j\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:51.409147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409092 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/799e2b87-1954-4103-860d-33ab42170349-trusted-ca\") pod \"console-operator-565b7b569d-sf2mn\" (UID: \"799e2b87-1954-4103-860d-33ab42170349\") " pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:07:51.409147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409128 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-tmp-dir\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:51.409457 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409155 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:07:51.409457 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409183 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h8jf\" (UniqueName: \"kubernetes.io/projected/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-kube-api-access-8h8jf\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.409457 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409216 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rvvd\" (UniqueName: \"kubernetes.io/projected/957c1e60-c08f-437f-afaf-d3679b2e8376-kube-api-access-8rvvd\") pod \"kube-storage-version-migrator-operator-7658d9b749-6s889\" (UID: \"957c1e60-c08f-437f-afaf-d3679b2e8376\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" Jul 22 16:07:51.409457 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409243 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.409457 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409274 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:51.409457 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409278 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-ca-trust-extracted\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.409457 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409361 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-image-registry-private-configuration\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.409457 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409391 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv95p\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-kube-api-access-dv95p\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.409457 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409422 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/90abd0b8-4b88-4da8-838f-3202f3d8ad13-serving-cert\") pod \"service-ca-operator-5bcbc94c78-zv9r5\" (UID: \"90abd0b8-4b88-4da8-838f-3202f3d8ad13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" Jul 22 16:07:51.409457 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409452 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:51.409826 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409475 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/957c1e60-c08f-437f-afaf-d3679b2e8376-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-6s889\" (UID: \"957c1e60-c08f-437f-afaf-d3679b2e8376\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" Jul 22 16:07:51.409826 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409505 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bec7900e-a6b5-4e09-b908-690e4d13b982-cert\") pod \"ingress-canary-vprx2\" (UID: \"bec7900e-a6b5-4e09-b908-690e4d13b982\") " pod="openshift-ingress-canary/ingress-canary-vprx2" Jul 22 16:07:51.409826 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409550 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/799e2b87-1954-4103-860d-33ab42170349-serving-cert\") pod \"console-operator-565b7b569d-sf2mn\" (UID: \"799e2b87-1954-4103-860d-33ab42170349\") " pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:07:51.409826 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409613 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.409826 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409649 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-certificates\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.409826 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409746 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-trusted-ca\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.409826 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409756 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks2q4\" (UID: \"7e88090f-ac10-4518-954c-aa515ceb5cb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" Jul 22 16:07:51.409826 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.409749 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:07:51.409826 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.409784 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b465bb546-ppslg: secret "image-registry-tls" not found Jul 22 16:07:51.409826 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409799 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdv9j\" (UniqueName: \"kubernetes.io/projected/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-kube-api-access-fdv9j\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:07:51.409826 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409825 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-tmp\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.410237 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.409855 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls podName:1fe0626b-4d62-4845-b99c-9ff8d9e711e5 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:51.909838635 +0000 UTC m=+37.051712705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls") pod "image-registry-6b465bb546-ppslg" (UID: "1fe0626b-4d62-4845-b99c-9ff8d9e711e5") : secret "image-registry-tls" not found Jul 22 16:07:51.410237 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409871 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnfcm\" (UniqueName: \"kubernetes.io/projected/799e2b87-1954-4103-860d-33ab42170349-kube-api-access-lnfcm\") pod \"console-operator-565b7b569d-sf2mn\" (UID: \"799e2b87-1954-4103-860d-33ab42170349\") " pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:07:51.410237 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409896 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/90abd0b8-4b88-4da8-838f-3202f3d8ad13-config\") pod \"service-ca-operator-5bcbc94c78-zv9r5\" (UID: \"90abd0b8-4b88-4da8-838f-3202f3d8ad13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" Jul 22 16:07:51.410237 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409915 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-service-ca-bundle\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.410237 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409939 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-installation-pull-secrets\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.410237 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409960 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:07:51.410237 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.409983 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvzkb\" (UniqueName: \"kubernetes.io/projected/bec7900e-a6b5-4e09-b908-690e4d13b982-kube-api-access-lvzkb\") pod \"ingress-canary-vprx2\" (UID: \"bec7900e-a6b5-4e09-b908-690e4d13b982\") " pod="openshift-ingress-canary/ingress-canary-vprx2" Jul 22 16:07:51.410237 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.410034 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh4dm\" (UniqueName: \"kubernetes.io/projected/90abd0b8-4b88-4da8-838f-3202f3d8ad13-kube-api-access-vh4dm\") pod \"service-ca-operator-5bcbc94c78-zv9r5\" (UID: \"90abd0b8-4b88-4da8-838f-3202f3d8ad13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" Jul 22 16:07:51.410237 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.410076 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-snapshots\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.410237 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.410111 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-certificates\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.417376 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.417348 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-image-registry-private-configuration\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.417562 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.417360 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-installation-pull-secrets\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.419795 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.419770 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-bound-sa-token\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.419949 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.419921 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv95p\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-kube-api-access-dv95p\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.511311 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511273 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-snapshots\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.511441 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511317 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6wk5\" (UniqueName: \"kubernetes.io/projected/dc08ab36-8b95-4b9f-a9dc-e3aade431059-kube-api-access-r6wk5\") pod \"volume-data-source-validator-78cf4c6f6-ltpvx\" (UID: \"dc08ab36-8b95-4b9f-a9dc-e3aade431059\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ltpvx" Jul 22 16:07:51.511514 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511445 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-default-certificate\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:51.511514 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511479 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/957c1e60-c08f-437f-afaf-d3679b2e8376-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-6s889\" (UID: \"957c1e60-c08f-437f-afaf-d3679b2e8376\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" Jul 22 16:07:51.511624 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511597 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:51.511717 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511629 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-stats-auth\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:51.511717 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511656 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-serving-cert\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.511717 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511709 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbnbm\" (UniqueName: \"kubernetes.io/projected/7e88090f-ac10-4518-954c-aa515ceb5cb7-kube-api-access-wbnbm\") pod \"cluster-samples-operator-79b7c55f87-ks2q4\" (UID: \"7e88090f-ac10-4518-954c-aa515ceb5cb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" Jul 22 16:07:51.511858 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511743 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w2gb\" (UniqueName: \"kubernetes.io/projected/57dbd4ff-f4ed-484a-801b-9cef8a961409-kube-api-access-4w2gb\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:51.511858 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.511746 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:07:51.511858 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511767 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/799e2b87-1954-4103-860d-33ab42170349-config\") pod \"console-operator-565b7b569d-sf2mn\" (UID: \"799e2b87-1954-4103-860d-33ab42170349\") " pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:07:51.511858 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511797 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94sgl\" (UniqueName: \"kubernetes.io/projected/0a773bcb-61a5-42df-a42c-31f53a91d776-kube-api-access-94sgl\") pod \"network-check-source-7bc5b6bf85-9nf8d\" (UID: \"0a773bcb-61a5-42df-a42c-31f53a91d776\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-9nf8d" Jul 22 16:07:51.511858 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.511823 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls podName:db5fa8ab-0e89-4823-9f94-353bdf81e0dc nodeName:}" failed. No retries permitted until 2026-07-22 16:07:52.011801879 +0000 UTC m=+37.153675938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls") pod "dns-default-c78v9" (UID: "db5fa8ab-0e89-4823-9f94-353bdf81e0dc") : secret "dns-default-metrics-tls" not found Jul 22 16:07:51.512119 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511864 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-config-volume\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:51.512119 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511911 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rv6j\" (UniqueName: \"kubernetes.io/projected/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-kube-api-access-2rv6j\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:51.512119 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511927 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-snapshots\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.512119 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.511935 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/799e2b87-1954-4103-860d-33ab42170349-trusted-ca\") pod \"console-operator-565b7b569d-sf2mn\" (UID: \"799e2b87-1954-4103-860d-33ab42170349\") " pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:07:51.512333 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512159 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-tmp-dir\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:51.512333 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512187 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:07:51.512333 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512213 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h8jf\" (UniqueName: \"kubernetes.io/projected/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-kube-api-access-8h8jf\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.512333 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512243 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rvvd\" (UniqueName: \"kubernetes.io/projected/957c1e60-c08f-437f-afaf-d3679b2e8376-kube-api-access-8rvvd\") pod \"kube-storage-version-migrator-operator-7658d9b749-6s889\" (UID: \"957c1e60-c08f-437f-afaf-d3679b2e8376\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" Jul 22 16:07:51.512333 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512274 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.512333 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512298 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:51.512594 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512348 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/90abd0b8-4b88-4da8-838f-3202f3d8ad13-serving-cert\") pod \"service-ca-operator-5bcbc94c78-zv9r5\" (UID: \"90abd0b8-4b88-4da8-838f-3202f3d8ad13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" Jul 22 16:07:51.512594 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512378 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:51.512594 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512404 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/957c1e60-c08f-437f-afaf-d3679b2e8376-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-6s889\" (UID: \"957c1e60-c08f-437f-afaf-d3679b2e8376\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" Jul 22 16:07:51.512594 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512427 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-tmp-dir\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:51.512594 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512435 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bec7900e-a6b5-4e09-b908-690e4d13b982-cert\") pod \"ingress-canary-vprx2\" (UID: \"bec7900e-a6b5-4e09-b908-690e4d13b982\") " pod="openshift-ingress-canary/ingress-canary-vprx2" Jul 22 16:07:51.512594 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512460 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/799e2b87-1954-4103-860d-33ab42170349-serving-cert\") pod \"console-operator-565b7b569d-sf2mn\" (UID: \"799e2b87-1954-4103-860d-33ab42170349\") " pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:07:51.512594 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512502 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks2q4\" (UID: \"7e88090f-ac10-4518-954c-aa515ceb5cb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" Jul 22 16:07:51.512594 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512528 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdv9j\" (UniqueName: \"kubernetes.io/projected/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-kube-api-access-fdv9j\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:07:51.512594 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.512549 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle podName:57dbd4ff-f4ed-484a-801b-9cef8a961409 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:52.012530771 +0000 UTC m=+37.154404843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle") pod "router-default-586bddb798-57m2m" (UID: "57dbd4ff-f4ed-484a-801b-9cef8a961409") : configmap references non-existent config key: service-ca.crt Jul 22 16:07:51.512594 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512579 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-tmp\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.513113 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512608 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnfcm\" (UniqueName: \"kubernetes.io/projected/799e2b87-1954-4103-860d-33ab42170349-kube-api-access-lnfcm\") pod \"console-operator-565b7b569d-sf2mn\" (UID: \"799e2b87-1954-4103-860d-33ab42170349\") " pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:07:51.513113 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512636 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/90abd0b8-4b88-4da8-838f-3202f3d8ad13-config\") pod \"service-ca-operator-5bcbc94c78-zv9r5\" (UID: \"90abd0b8-4b88-4da8-838f-3202f3d8ad13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" Jul 22 16:07:51.513113 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512657 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-config-volume\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:51.513113 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512662 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-service-ca-bundle\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.513113 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512721 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:07:51.513113 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512750 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvzkb\" (UniqueName: \"kubernetes.io/projected/bec7900e-a6b5-4e09-b908-690e4d13b982-kube-api-access-lvzkb\") pod \"ingress-canary-vprx2\" (UID: \"bec7900e-a6b5-4e09-b908-690e4d13b982\") " pod="openshift-ingress-canary/ingress-canary-vprx2" Jul 22 16:07:51.513113 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512788 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh4dm\" (UniqueName: \"kubernetes.io/projected/90abd0b8-4b88-4da8-838f-3202f3d8ad13-kube-api-access-vh4dm\") pod \"service-ca-operator-5bcbc94c78-zv9r5\" (UID: \"90abd0b8-4b88-4da8-838f-3202f3d8ad13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" Jul 22 16:07:51.513113 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512942 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/799e2b87-1954-4103-860d-33ab42170349-trusted-ca\") pod \"console-operator-565b7b569d-sf2mn\" (UID: \"799e2b87-1954-4103-860d-33ab42170349\") " pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:07:51.513113 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.512948 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/799e2b87-1954-4103-860d-33ab42170349-config\") pod \"console-operator-565b7b569d-sf2mn\" (UID: \"799e2b87-1954-4103-860d-33ab42170349\") " pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:07:51.513562 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.513126 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:07:51.513562 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.513183 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls podName:fa7a9636-f842-48f7-ae45-dfcca3c01ee3 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:52.01316677 +0000 UTC m=+37.155040825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rg4vq" (UID: "fa7a9636-f842-48f7-ae45-dfcca3c01ee3") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:07:51.513562 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.513260 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 16:07:51.513562 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.513279 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-service-ca-bundle\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.513562 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.513305 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs podName:57dbd4ff-f4ed-484a-801b-9cef8a961409 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:52.013290558 +0000 UTC m=+37.155164615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs") pod "router-default-586bddb798-57m2m" (UID: "57dbd4ff-f4ed-484a-801b-9cef8a961409") : secret "router-metrics-certs-default" not found Jul 22 16:07:51.513863 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.513629 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:07:51.514399 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.514044 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-tmp\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.514399 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.514134 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:07:51.514399 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.514163 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/90abd0b8-4b88-4da8-838f-3202f3d8ad13-config\") pod \"service-ca-operator-5bcbc94c78-zv9r5\" (UID: \"90abd0b8-4b88-4da8-838f-3202f3d8ad13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" Jul 22 16:07:51.514399 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.514219 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls podName:7e88090f-ac10-4518-954c-aa515ceb5cb7 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:52.014203706 +0000 UTC m=+37.156077764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ks2q4" (UID: "7e88090f-ac10-4518-954c-aa515ceb5cb7") : secret "samples-operator-tls" not found Jul 22 16:07:51.514658 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.514401 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.514658 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.514437 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/957c1e60-c08f-437f-afaf-d3679b2e8376-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-6s889\" (UID: \"957c1e60-c08f-437f-afaf-d3679b2e8376\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" Jul 22 16:07:51.514658 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.514632 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-stats-auth\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:51.515601 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.515578 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-serving-cert\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.515601 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.515596 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/90abd0b8-4b88-4da8-838f-3202f3d8ad13-serving-cert\") pod \"service-ca-operator-5bcbc94c78-zv9r5\" (UID: \"90abd0b8-4b88-4da8-838f-3202f3d8ad13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" Jul 22 16:07:51.515946 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.515930 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/799e2b87-1954-4103-860d-33ab42170349-serving-cert\") pod \"console-operator-565b7b569d-sf2mn\" (UID: \"799e2b87-1954-4103-860d-33ab42170349\") " pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:07:51.516022 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.516002 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/957c1e60-c08f-437f-afaf-d3679b2e8376-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-6s889\" (UID: \"957c1e60-c08f-437f-afaf-d3679b2e8376\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" Jul 22 16:07:51.516297 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.516278 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-default-certificate\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:51.516454 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.516438 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bec7900e-a6b5-4e09-b908-690e4d13b982-cert\") pod \"ingress-canary-vprx2\" (UID: \"bec7900e-a6b5-4e09-b908-690e4d13b982\") " pod="openshift-ingress-canary/ingress-canary-vprx2" Jul 22 16:07:51.522120 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.521773 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbnbm\" (UniqueName: \"kubernetes.io/projected/7e88090f-ac10-4518-954c-aa515ceb5cb7-kube-api-access-wbnbm\") pod \"cluster-samples-operator-79b7c55f87-ks2q4\" (UID: \"7e88090f-ac10-4518-954c-aa515ceb5cb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" Jul 22 16:07:51.522120 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.521864 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6wk5\" (UniqueName: \"kubernetes.io/projected/dc08ab36-8b95-4b9f-a9dc-e3aade431059-kube-api-access-r6wk5\") pod \"volume-data-source-validator-78cf4c6f6-ltpvx\" (UID: \"dc08ab36-8b95-4b9f-a9dc-e3aade431059\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ltpvx" Jul 22 16:07:51.522120 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.522017 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94sgl\" (UniqueName: \"kubernetes.io/projected/0a773bcb-61a5-42df-a42c-31f53a91d776-kube-api-access-94sgl\") pod \"network-check-source-7bc5b6bf85-9nf8d\" (UID: \"0a773bcb-61a5-42df-a42c-31f53a91d776\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-9nf8d" Jul 22 16:07:51.522335 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.522275 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w2gb\" (UniqueName: \"kubernetes.io/projected/57dbd4ff-f4ed-484a-801b-9cef8a961409-kube-api-access-4w2gb\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:51.522386 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.522336 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdv9j\" (UniqueName: \"kubernetes.io/projected/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-kube-api-access-fdv9j\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:07:51.523522 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.523503 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnfcm\" (UniqueName: \"kubernetes.io/projected/799e2b87-1954-4103-860d-33ab42170349-kube-api-access-lnfcm\") pod \"console-operator-565b7b569d-sf2mn\" (UID: \"799e2b87-1954-4103-860d-33ab42170349\") " pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:07:51.523681 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.523660 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rv6j\" (UniqueName: \"kubernetes.io/projected/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-kube-api-access-2rv6j\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:51.524165 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.524142 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h8jf\" (UniqueName: \"kubernetes.io/projected/0f6ccc41-e229-40a9-9a3a-a6fa628dfe86-kube-api-access-8h8jf\") pod \"insights-operator-555d9d7885-24cbv\" (UID: \"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86\") " pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.524261 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.524240 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rvvd\" (UniqueName: \"kubernetes.io/projected/957c1e60-c08f-437f-afaf-d3679b2e8376-kube-api-access-8rvvd\") pod \"kube-storage-version-migrator-operator-7658d9b749-6s889\" (UID: \"957c1e60-c08f-437f-afaf-d3679b2e8376\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" Jul 22 16:07:51.524294 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.524255 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvzkb\" (UniqueName: \"kubernetes.io/projected/bec7900e-a6b5-4e09-b908-690e4d13b982-kube-api-access-lvzkb\") pod \"ingress-canary-vprx2\" (UID: \"bec7900e-a6b5-4e09-b908-690e4d13b982\") " pod="openshift-ingress-canary/ingress-canary-vprx2" Jul 22 16:07:51.524294 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.524251 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh4dm\" (UniqueName: \"kubernetes.io/projected/90abd0b8-4b88-4da8-838f-3202f3d8ad13-kube-api-access-vh4dm\") pod \"service-ca-operator-5bcbc94c78-zv9r5\" (UID: \"90abd0b8-4b88-4da8-838f-3202f3d8ad13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" Jul 22 16:07:51.549831 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.549800 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-24cbv" Jul 22 16:07:51.558205 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.558183 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ltpvx" Jul 22 16:07:51.566200 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.566044 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" Jul 22 16:07:51.582857 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.582817 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" Jul 22 16:07:51.591971 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.591939 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:07:51.599378 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.599342 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-9nf8d" Jul 22 16:07:51.603443 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.603418 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vprx2" Jul 22 16:07:51.750014 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.749950 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ltpvx"] Jul 22 16:07:51.755968 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:51.755926 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc08ab36_8b95_4b9f_a9dc_e3aade431059.slice/crio-92e9e9012153ac2ac722d5cfd804e28882c36cddec35c815ce44953e6cfb4679 WatchSource:0}: Error finding container 92e9e9012153ac2ac722d5cfd804e28882c36cddec35c815ce44953e6cfb4679: Status 404 returned error can't find the container with id 92e9e9012153ac2ac722d5cfd804e28882c36cddec35c815ce44953e6cfb4679 Jul 22 16:07:51.769511 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.769442 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5"] Jul 22 16:07:51.775729 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.775651 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-24cbv"] Jul 22 16:07:51.784487 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:51.784458 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6ccc41_e229_40a9_9a3a_a6fa628dfe86.slice/crio-c68c06f3684b057a0567f7828d2ce21ef4ca0e38c32ad3df3fd4b909cc656045 WatchSource:0}: Error finding container c68c06f3684b057a0567f7828d2ce21ef4ca0e38c32ad3df3fd4b909cc656045: Status 404 returned error can't find the container with id c68c06f3684b057a0567f7828d2ce21ef4ca0e38c32ad3df3fd4b909cc656045 Jul 22 16:07:51.798118 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.798069 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-sf2mn"] Jul 22 16:07:51.800720 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.800700 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889"] Jul 22 16:07:51.801781 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:51.801751 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799e2b87_1954_4103_860d_33ab42170349.slice/crio-340a39f2fe475b5878658cbb73a404a8cf3c51b84df58b209b9000b78ccdb8b3 WatchSource:0}: Error finding container 340a39f2fe475b5878658cbb73a404a8cf3c51b84df58b209b9000b78ccdb8b3: Status 404 returned error can't find the container with id 340a39f2fe475b5878658cbb73a404a8cf3c51b84df58b209b9000b78ccdb8b3 Jul 22 16:07:51.808850 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:51.808790 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957c1e60_c08f_437f_afaf_d3679b2e8376.slice/crio-593730615d3d3834ed333eb847a45dc672aea607e544407a4fe38f91da8a2534 WatchSource:0}: Error finding container 593730615d3d3834ed333eb847a45dc672aea607e544407a4fe38f91da8a2534: Status 404 returned error can't find the container with id 593730615d3d3834ed333eb847a45dc672aea607e544407a4fe38f91da8a2534 Jul 22 16:07:51.809831 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.809810 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-9nf8d"] Jul 22 16:07:51.812495 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:51.812473 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a773bcb_61a5_42df_a42c_31f53a91d776.slice/crio-5e5b0b20a2c89539fab0dae4cb935727af74dab4626efccae1248fde97273af0 WatchSource:0}: Error finding container 5e5b0b20a2c89539fab0dae4cb935727af74dab4626efccae1248fde97273af0: Status 404 returned error can't find the container with id 5e5b0b20a2c89539fab0dae4cb935727af74dab4626efccae1248fde97273af0 Jul 22 16:07:51.826063 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.826044 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vprx2"] Jul 22 16:07:51.828575 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:07:51.828550 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec7900e_a6b5_4e09_b908_690e4d13b982.slice/crio-79b1538f65c3aa778823ac16c69f4aeae73be8c1dae028c9889a4291ee39fc9c WatchSource:0}: Error finding container 79b1538f65c3aa778823ac16c69f4aeae73be8c1dae028c9889a4291ee39fc9c: Status 404 returned error can't find the container with id 79b1538f65c3aa778823ac16c69f4aeae73be8c1dae028c9889a4291ee39fc9c Jul 22 16:07:51.918991 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.918909 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:51.919130 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.919045 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:07:51.919130 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.919062 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b465bb546-ppslg: secret "image-registry-tls" not found Jul 22 16:07:51.919130 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:51.919114 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls podName:1fe0626b-4d62-4845-b99c-9ff8d9e711e5 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:52.919099058 +0000 UTC m=+38.060973114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls") pod "image-registry-6b465bb546-ppslg" (UID: "1fe0626b-4d62-4845-b99c-9ff8d9e711e5") : secret "image-registry-tls" not found Jul 22 16:07:51.977725 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.977678 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:07:51.977725 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.977713 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:07:51.977909 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.977731 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:07:51.980833 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.980813 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 16:07:51.982239 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.982224 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rb5n6\"" Jul 22 16:07:51.985831 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.984585 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 16:07:51.985831 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:51.984903 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s68m6\"" Jul 22 16:07:52.020191 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.020162 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:52.020327 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.020210 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:52.020327 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.020238 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks2q4\" (UID: \"7e88090f-ac10-4518-954c-aa515ceb5cb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" Jul 22 16:07:52.020327 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.020262 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:07:52.020327 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.020291 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:52.020327 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:52.020316 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle podName:57dbd4ff-f4ed-484a-801b-9cef8a961409 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:53.020298641 +0000 UTC m=+38.162172696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle") pod "router-default-586bddb798-57m2m" (UID: "57dbd4ff-f4ed-484a-801b-9cef8a961409") : configmap references non-existent config key: service-ca.crt Jul 22 16:07:52.020534 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:52.020381 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:07:52.020534 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:52.020384 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:07:52.020534 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:52.020395 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 16:07:52.020534 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:52.020418 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:07:52.020534 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:52.020423 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls podName:db5fa8ab-0e89-4823-9f94-353bdf81e0dc nodeName:}" failed. No retries permitted until 2026-07-22 16:07:53.020409684 +0000 UTC m=+38.162283742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls") pod "dns-default-c78v9" (UID: "db5fa8ab-0e89-4823-9f94-353bdf81e0dc") : secret "dns-default-metrics-tls" not found Jul 22 16:07:52.020534 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:52.020476 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls podName:7e88090f-ac10-4518-954c-aa515ceb5cb7 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:53.020463899 +0000 UTC m=+38.162337954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ks2q4" (UID: "7e88090f-ac10-4518-954c-aa515ceb5cb7") : secret "samples-operator-tls" not found Jul 22 16:07:52.020534 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:52.020491 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs podName:57dbd4ff-f4ed-484a-801b-9cef8a961409 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:53.020482718 +0000 UTC m=+38.162356773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs") pod "router-default-586bddb798-57m2m" (UID: "57dbd4ff-f4ed-484a-801b-9cef8a961409") : secret "router-metrics-certs-default" not found Jul 22 16:07:52.020534 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:52.020509 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls podName:fa7a9636-f842-48f7-ae45-dfcca3c01ee3 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:53.020502949 +0000 UTC m=+38.162377004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rg4vq" (UID: "fa7a9636-f842-48f7-ae45-dfcca3c01ee3") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:07:52.132901 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.132862 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" event={"ID":"957c1e60-c08f-437f-afaf-d3679b2e8376","Type":"ContainerStarted","Data":"593730615d3d3834ed333eb847a45dc672aea607e544407a4fe38f91da8a2534"} Jul 22 16:07:52.133864 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.133827 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-24cbv" event={"ID":"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86","Type":"ContainerStarted","Data":"c68c06f3684b057a0567f7828d2ce21ef4ca0e38c32ad3df3fd4b909cc656045"} Jul 22 16:07:52.136191 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.136169 2567 generic.go:356] "Generic (PLEG): container finished" podID="6052e9bc-e447-449b-86eb-d4484c381c9c" containerID="f449a8026875472f15e2a407be44d55ad24e46ae310da5b7ac1c2ff0455dedc6" exitCode=0 Jul 22 16:07:52.136319 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.136228 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq6wn" event={"ID":"6052e9bc-e447-449b-86eb-d4484c381c9c","Type":"ContainerDied","Data":"f449a8026875472f15e2a407be44d55ad24e46ae310da5b7ac1c2ff0455dedc6"} Jul 22 16:07:52.137349 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.137320 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" event={"ID":"799e2b87-1954-4103-860d-33ab42170349","Type":"ContainerStarted","Data":"340a39f2fe475b5878658cbb73a404a8cf3c51b84df58b209b9000b78ccdb8b3"} Jul 22 16:07:52.138362 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.138340 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" event={"ID":"90abd0b8-4b88-4da8-838f-3202f3d8ad13","Type":"ContainerStarted","Data":"96d8f8b2fd3bf4e393ea8ed92ada9bc59da0f39d691ae6d5d03fc48cc086db04"} Jul 22 16:07:52.139372 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.139345 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ltpvx" event={"ID":"dc08ab36-8b95-4b9f-a9dc-e3aade431059","Type":"ContainerStarted","Data":"92e9e9012153ac2ac722d5cfd804e28882c36cddec35c815ce44953e6cfb4679"} Jul 22 16:07:52.140345 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.140320 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vprx2" event={"ID":"bec7900e-a6b5-4e09-b908-690e4d13b982","Type":"ContainerStarted","Data":"79b1538f65c3aa778823ac16c69f4aeae73be8c1dae028c9889a4291ee39fc9c"} Jul 22 16:07:52.141327 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.141308 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-9nf8d" event={"ID":"0a773bcb-61a5-42df-a42c-31f53a91d776","Type":"ContainerStarted","Data":"5e5b0b20a2c89539fab0dae4cb935727af74dab4626efccae1248fde97273af0"} Jul 22 16:07:52.930225 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:52.930184 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:52.930800 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:52.930410 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:07:52.930800 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:52.930466 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b465bb546-ppslg: secret "image-registry-tls" not found Jul 22 16:07:52.930800 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:52.930527 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls podName:1fe0626b-4d62-4845-b99c-9ff8d9e711e5 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:54.930508425 +0000 UTC m=+40.072382483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls") pod "image-registry-6b465bb546-ppslg" (UID: "1fe0626b-4d62-4845-b99c-9ff8d9e711e5") : secret "image-registry-tls" not found Jul 22 16:07:53.032724 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:53.031714 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:53.032724 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:53.031814 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:53.032724 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:53.031868 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:53.032724 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:53.031913 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks2q4\" (UID: \"7e88090f-ac10-4518-954c-aa515ceb5cb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" Jul 22 16:07:53.032724 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:53.031948 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:07:53.032724 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:53.032087 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:07:53.032724 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:53.032145 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls podName:fa7a9636-f842-48f7-ae45-dfcca3c01ee3 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:55.032127796 +0000 UTC m=+40.174001856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rg4vq" (UID: "fa7a9636-f842-48f7-ae45-dfcca3c01ee3") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:07:53.032724 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:53.032209 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:07:53.032724 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:53.032240 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls podName:db5fa8ab-0e89-4823-9f94-353bdf81e0dc nodeName:}" failed. No retries permitted until 2026-07-22 16:07:55.032229019 +0000 UTC m=+40.174103080 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls") pod "dns-default-c78v9" (UID: "db5fa8ab-0e89-4823-9f94-353bdf81e0dc") : secret "dns-default-metrics-tls" not found Jul 22 16:07:53.032724 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:53.032301 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle podName:57dbd4ff-f4ed-484a-801b-9cef8a961409 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:55.032291491 +0000 UTC m=+40.174165549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle") pod "router-default-586bddb798-57m2m" (UID: "57dbd4ff-f4ed-484a-801b-9cef8a961409") : configmap references non-existent config key: service-ca.crt Jul 22 16:07:53.032724 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:53.032350 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 16:07:53.032724 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:53.032390 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs podName:57dbd4ff-f4ed-484a-801b-9cef8a961409 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:55.032380808 +0000 UTC m=+40.174254866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs") pod "router-default-586bddb798-57m2m" (UID: "57dbd4ff-f4ed-484a-801b-9cef8a961409") : secret "router-metrics-certs-default" not found Jul 22 16:07:53.032724 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:53.032440 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:07:53.032724 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:53.032468 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls podName:7e88090f-ac10-4518-954c-aa515ceb5cb7 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:55.032458364 +0000 UTC m=+40.174332424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ks2q4" (UID: "7e88090f-ac10-4518-954c-aa515ceb5cb7") : secret "samples-operator-tls" not found Jul 22 16:07:53.161371 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:53.161050 2567 generic.go:356] "Generic (PLEG): container finished" podID="6052e9bc-e447-449b-86eb-d4484c381c9c" containerID="fc2b76887dedf99d750dfbb1180b23d1b3a03b2bc49c29d3ef0c2161a9c840d5" exitCode=0 Jul 22 16:07:53.161371 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:53.161217 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq6wn" event={"ID":"6052e9bc-e447-449b-86eb-d4484c381c9c","Type":"ContainerDied","Data":"fc2b76887dedf99d750dfbb1180b23d1b3a03b2bc49c29d3ef0c2161a9c840d5"} Jul 22 16:07:54.170304 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:54.170206 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq6wn" event={"ID":"6052e9bc-e447-449b-86eb-d4484c381c9c","Type":"ContainerStarted","Data":"eac79ce00d68e7d101014cad8fa3994738be72a90a2e4f756f584e7db87e2aea"} Jul 22 16:07:54.199303 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:54.199176 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xq6wn" podStartSLOduration=9.214978553 podStartE2EDuration="38.199159332s" totalImagesPullingTime="28.984180779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:07:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:07:54.198975931 +0000 UTC m=+39.340850020" watchObservedRunningTime="2026-07-22 16:07:54.199159332 +0000 UTC m=+39.341033408" Jul 22 16:07:54.956522 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:54.956481 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:54.956811 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:54.956650 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:07:54.956811 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:54.956674 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b465bb546-ppslg: secret "image-registry-tls" not found Jul 22 16:07:54.956811 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:54.956771 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls podName:1fe0626b-4d62-4845-b99c-9ff8d9e711e5 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:58.956742732 +0000 UTC m=+44.098616804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls") pod "image-registry-6b465bb546-ppslg" (UID: "1fe0626b-4d62-4845-b99c-9ff8d9e711e5") : secret "image-registry-tls" not found Jul 22 16:07:55.057158 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:55.057121 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks2q4\" (UID: \"7e88090f-ac10-4518-954c-aa515ceb5cb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" Jul 22 16:07:55.057339 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:55.057176 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:07:55.057339 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:55.057215 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:55.057339 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:55.057273 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:07:55.057339 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:55.057330 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:07:55.057549 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:55.057350 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls podName:7e88090f-ac10-4518-954c-aa515ceb5cb7 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:59.057329952 +0000 UTC m=+44.199204028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ks2q4" (UID: "7e88090f-ac10-4518-954c-aa515ceb5cb7") : secret "samples-operator-tls" not found Jul 22 16:07:55.057549 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:55.057282 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:55.057549 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:55.057368 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls podName:fa7a9636-f842-48f7-ae45-dfcca3c01ee3 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:59.057358079 +0000 UTC m=+44.199232146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rg4vq" (UID: "fa7a9636-f842-48f7-ae45-dfcca3c01ee3") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:07:55.057549 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:55.057384 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:07:55.057549 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:55.057389 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle podName:57dbd4ff-f4ed-484a-801b-9cef8a961409 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:59.057380708 +0000 UTC m=+44.199254768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle") pod "router-default-586bddb798-57m2m" (UID: "57dbd4ff-f4ed-484a-801b-9cef8a961409") : configmap references non-existent config key: service-ca.crt Jul 22 16:07:55.057549 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:55.057452 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls podName:db5fa8ab-0e89-4823-9f94-353bdf81e0dc nodeName:}" failed. No retries permitted until 2026-07-22 16:07:59.057436099 +0000 UTC m=+44.199310162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls") pod "dns-default-c78v9" (UID: "db5fa8ab-0e89-4823-9f94-353bdf81e0dc") : secret "dns-default-metrics-tls" not found Jul 22 16:07:55.057549 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:55.057511 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:55.057935 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:55.057599 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 16:07:55.057935 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:55.057634 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs podName:57dbd4ff-f4ed-484a-801b-9cef8a961409 nodeName:}" failed. No retries permitted until 2026-07-22 16:07:59.057623897 +0000 UTC m=+44.199497953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs") pod "router-default-586bddb798-57m2m" (UID: "57dbd4ff-f4ed-484a-801b-9cef8a961409") : secret "router-metrics-certs-default" not found Jul 22 16:07:58.995055 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:58.995011 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:07:58.995751 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:58.995165 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:07:58.995751 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:58.995184 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b465bb546-ppslg: secret "image-registry-tls" not found Jul 22 16:07:58.995751 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:58.995251 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls podName:1fe0626b-4d62-4845-b99c-9ff8d9e711e5 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:06.995230939 +0000 UTC m=+52.137105000 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls") pod "image-registry-6b465bb546-ppslg" (UID: "1fe0626b-4d62-4845-b99c-9ff8d9e711e5") : secret "image-registry-tls" not found Jul 22 16:07:59.096482 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:59.096444 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks2q4\" (UID: \"7e88090f-ac10-4518-954c-aa515ceb5cb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" Jul 22 16:07:59.096482 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:59.096486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:07:59.096719 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:59.096517 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:07:59.096719 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:59.096573 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:59.096719 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:59.096596 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:07:59.096719 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:07:59.096612 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:07:59.096719 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:59.096617 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:07:59.096719 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:59.096671 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls podName:fa7a9636-f842-48f7-ae45-dfcca3c01ee3 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:07.096651103 +0000 UTC m=+52.238525172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rg4vq" (UID: "fa7a9636-f842-48f7-ae45-dfcca3c01ee3") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:07:59.096719 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:59.096681 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 16:07:59.096719 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:59.096710 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:07:59.096993 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:59.096686 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls podName:7e88090f-ac10-4518-954c-aa515ceb5cb7 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:07.096679187 +0000 UTC m=+52.238553242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ks2q4" (UID: "7e88090f-ac10-4518-954c-aa515ceb5cb7") : secret "samples-operator-tls" not found Jul 22 16:07:59.096993 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:59.096754 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs podName:57dbd4ff-f4ed-484a-801b-9cef8a961409 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:07.096743959 +0000 UTC m=+52.238618014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs") pod "router-default-586bddb798-57m2m" (UID: "57dbd4ff-f4ed-484a-801b-9cef8a961409") : secret "router-metrics-certs-default" not found Jul 22 16:07:59.096993 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:59.096775 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls podName:db5fa8ab-0e89-4823-9f94-353bdf81e0dc nodeName:}" failed. No retries permitted until 2026-07-22 16:08:07.096767673 +0000 UTC m=+52.238641732 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls") pod "dns-default-c78v9" (UID: "db5fa8ab-0e89-4823-9f94-353bdf81e0dc") : secret "dns-default-metrics-tls" not found Jul 22 16:07:59.096993 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:07:59.096790 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle podName:57dbd4ff-f4ed-484a-801b-9cef8a961409 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:07.096781923 +0000 UTC m=+52.238655978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle") pod "router-default-586bddb798-57m2m" (UID: "57dbd4ff-f4ed-484a-801b-9cef8a961409") : configmap references non-existent config key: service-ca.crt Jul 22 16:08:00.194014 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.193933 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-9nf8d" event={"ID":"0a773bcb-61a5-42df-a42c-31f53a91d776","Type":"ContainerStarted","Data":"d7be7a67069c45afb0660753fa3954c1b241885407416210777923ffb3f49b4a"} Jul 22 16:08:00.195465 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.195437 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" event={"ID":"957c1e60-c08f-437f-afaf-d3679b2e8376","Type":"ContainerStarted","Data":"055f78c9bd24e17d138701ff4fb1af7aa782ae1a7a0f8a9a545d3f3598f924b2"} Jul 22 16:08:00.197188 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.197144 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-24cbv" event={"ID":"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86","Type":"ContainerStarted","Data":"0a784101e3ecc4ca5c3331499a04e1b3995e09985dae7ead6ebb28e4a9746ae6"} Jul 22 16:08:00.198981 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.198961 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/0.log" Jul 22 16:08:00.199081 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.198995 2567 generic.go:356] "Generic (PLEG): container finished" podID="799e2b87-1954-4103-860d-33ab42170349" containerID="d8d9616b3fa23f76be6c243a6406316e562d61381fb24a8a1241bfa7f4ef2cc3" exitCode=255 Jul 22 16:08:00.199220 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.199194 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" event={"ID":"799e2b87-1954-4103-860d-33ab42170349","Type":"ContainerDied","Data":"d8d9616b3fa23f76be6c243a6406316e562d61381fb24a8a1241bfa7f4ef2cc3"} Jul 22 16:08:00.199337 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.199321 2567 scope.go:117] "RemoveContainer" containerID="d8d9616b3fa23f76be6c243a6406316e562d61381fb24a8a1241bfa7f4ef2cc3" Jul 22 16:08:00.200769 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.200720 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" event={"ID":"90abd0b8-4b88-4da8-838f-3202f3d8ad13","Type":"ContainerStarted","Data":"127d14a19301ae22c1ca51d5b7f8fb0abf0bb9df4e352924734899dd7939053c"} Jul 22 16:08:00.202372 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.202349 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ltpvx" event={"ID":"dc08ab36-8b95-4b9f-a9dc-e3aade431059","Type":"ContainerStarted","Data":"efd8195950849a2a0e5fcbeab302f4111949a27f0d9278cba22350369bba144b"} Jul 22 16:08:00.204302 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.204284 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vprx2" event={"ID":"bec7900e-a6b5-4e09-b908-690e4d13b982","Type":"ContainerStarted","Data":"6484a2e0665cb4fbe34951c4d24702eb88662b2df54aa59913b63cd2b28c3b5d"} Jul 22 16:08:00.236591 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.236547 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-9nf8d" podStartSLOduration=4.424132655 podStartE2EDuration="12.236528607s" totalImagesPullingTime="7.812395952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:00.214351897 +0000 UTC m=+45.356225975" watchObservedRunningTime="2026-07-22 16:08:00.236528607 +0000 UTC m=+45.378402685" Jul 22 16:08:00.237054 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.237021 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-24cbv" podStartSLOduration=22.399282204 podStartE2EDuration="30.237007957s" totalImagesPullingTime="7.837725753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:00.235804762 +0000 UTC m=+45.377678856" watchObservedRunningTime="2026-07-22 16:08:00.237007957 +0000 UTC m=+45.378882035" Jul 22 16:08:00.280386 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.280338 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ltpvx" podStartSLOduration=13.510921401000001 podStartE2EDuration="21.280319713s" totalImagesPullingTime="7.769398312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:00.279545303 +0000 UTC m=+45.421419406" watchObservedRunningTime="2026-07-22 16:08:00.280319713 +0000 UTC m=+45.422193790" Jul 22 16:08:00.308289 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.308248 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" podStartSLOduration=12.465547218 podStartE2EDuration="20.308229203s" totalImagesPullingTime="7.842681985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:00.307833956 +0000 UTC m=+45.449708033" watchObservedRunningTime="2026-07-22 16:08:00.308229203 +0000 UTC m=+45.450103281" Jul 22 16:08:00.369253 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.369197 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" podStartSLOduration=12.553618448 podStartE2EDuration="20.369173951s" totalImagesPullingTime="7.815555503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:00.339081428 +0000 UTC m=+45.480955507" watchObservedRunningTime="2026-07-22 16:08:00.369173951 +0000 UTC m=+45.511048029" Jul 22 16:08:00.369394 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:00.369296 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vprx2" podStartSLOduration=1.565484717 podStartE2EDuration="9.369289433s" totalImagesPullingTime="7.803804716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:00.368295821 +0000 UTC m=+45.510169900" watchObservedRunningTime="2026-07-22 16:08:00.369289433 +0000 UTC m=+45.511163511" Jul 22 16:08:01.187617 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.187578 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5bbfb"] Jul 22 16:08:01.209783 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.209749 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/1.log" Jul 22 16:08:01.210256 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.210237 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/0.log" Jul 22 16:08:01.210437 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.210421 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.210527 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.210337 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5bbfb"] Jul 22 16:08:01.210585 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.210421 2567 generic.go:356] "Generic (PLEG): container finished" podID="799e2b87-1954-4103-860d-33ab42170349" containerID="93c82489c9f35d1297898d2414a169ae3edfee51ac5229f02d7d2eee57aaaecf" exitCode=255 Jul 22 16:08:01.210585 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.210553 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" event={"ID":"799e2b87-1954-4103-860d-33ab42170349","Type":"ContainerDied","Data":"93c82489c9f35d1297898d2414a169ae3edfee51ac5229f02d7d2eee57aaaecf"} Jul 22 16:08:01.210713 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.210645 2567 scope.go:117] "RemoveContainer" containerID="d8d9616b3fa23f76be6c243a6406316e562d61381fb24a8a1241bfa7f4ef2cc3" Jul 22 16:08:01.210873 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.210851 2567 scope.go:117] "RemoveContainer" containerID="93c82489c9f35d1297898d2414a169ae3edfee51ac5229f02d7d2eee57aaaecf" Jul 22 16:08:01.211078 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:01.211052 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-sf2mn_openshift-console-operator(799e2b87-1954-4103-860d-33ab42170349)\"" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" podUID="799e2b87-1954-4103-860d-33ab42170349" Jul 22 16:08:01.215840 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.215540 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 16:08:01.215941 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.215877 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 16:08:01.217086 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.217065 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cvcbr\"" Jul 22 16:08:01.316553 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.316513 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.316788 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.316736 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51215a2f-bce7-4a3d-a206-d96885c4c724-crio-socket\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.316963 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.316915 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51215a2f-bce7-4a3d-a206-d96885c4c724-data-volume\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.317437 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.317415 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.317541 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.317514 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8vh\" (UniqueName: \"kubernetes.io/projected/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-api-access-vd8vh\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.418049 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.418003 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.418238 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.418065 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8vh\" (UniqueName: \"kubernetes.io/projected/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-api-access-vd8vh\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.418238 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.418160 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.418346 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:01.418161 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:08:01.418346 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.418229 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51215a2f-bce7-4a3d-a206-d96885c4c724-crio-socket\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.418346 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:01.418284 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:08:01.418463 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:01.418425 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls podName:51215a2f-bce7-4a3d-a206-d96885c4c724 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:01.918405269 +0000 UTC m=+47.060279338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bbfb" (UID: "51215a2f-bce7-4a3d-a206-d96885c4c724") : secret "insights-runtime-extractor-tls" not found Jul 22 16:08:01.418463 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:01.418443 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm podName:51215a2f-bce7-4a3d-a206-d96885c4c724 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:01.918434433 +0000 UTC m=+47.060308494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bbfb" (UID: "51215a2f-bce7-4a3d-a206-d96885c4c724") : configmap "kube-rbac-proxy" not found Jul 22 16:08:01.418566 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.418496 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51215a2f-bce7-4a3d-a206-d96885c4c724-crio-socket\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.418629 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.418599 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51215a2f-bce7-4a3d-a206-d96885c4c724-data-volume\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.418962 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.418944 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51215a2f-bce7-4a3d-a206-d96885c4c724-data-volume\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.429019 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.428997 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8vh\" (UniqueName: \"kubernetes.io/projected/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-api-access-vd8vh\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.592722 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.592672 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:08:01.592875 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.592743 2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:08:01.923151 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.923065 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.923293 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:01.923164 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:01.923293 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:01.923195 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:08:01.923367 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:01.923302 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm podName:51215a2f-bce7-4a3d-a206-d96885c4c724 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:02.923286972 +0000 UTC m=+48.065161028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bbfb" (UID: "51215a2f-bce7-4a3d-a206-d96885c4c724") : configmap "kube-rbac-proxy" not found Jul 22 16:08:01.923367 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:01.923321 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:08:01.923441 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:01.923379 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls podName:51215a2f-bce7-4a3d-a206-d96885c4c724 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:02.923362878 +0000 UTC m=+48.065236949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bbfb" (UID: "51215a2f-bce7-4a3d-a206-d96885c4c724") : secret "insights-runtime-extractor-tls" not found Jul 22 16:08:02.214666 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:02.214591 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/1.log" Jul 22 16:08:02.215034 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:02.214911 2567 scope.go:117] "RemoveContainer" containerID="93c82489c9f35d1297898d2414a169ae3edfee51ac5229f02d7d2eee57aaaecf" Jul 22 16:08:02.215071 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:02.215054 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-sf2mn_openshift-console-operator(799e2b87-1954-4103-860d-33ab42170349)\"" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" podUID="799e2b87-1954-4103-860d-33ab42170349" Jul 22 16:08:02.931716 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:02.931664 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:02.931884 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:02.931783 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:02.931884 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:02.931807 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:08:02.931967 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:02.931892 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm podName:51215a2f-bce7-4a3d-a206-d96885c4c724 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:04.93187042 +0000 UTC m=+50.073744478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bbfb" (UID: "51215a2f-bce7-4a3d-a206-d96885c4c724") : configmap "kube-rbac-proxy" not found Jul 22 16:08:02.931967 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:02.931894 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:08:02.931967 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:02.931934 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls podName:51215a2f-bce7-4a3d-a206-d96885c4c724 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:04.931922464 +0000 UTC m=+50.073796520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bbfb" (UID: "51215a2f-bce7-4a3d-a206-d96885c4c724") : secret "insights-runtime-extractor-tls" not found Jul 22 16:08:03.208792 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:03.208707 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j8z9t_808dd442-3437-4347-a145-3cf2f7315f6c/dns-node-resolver/0.log" Jul 22 16:08:03.217164 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:03.217138 2567 scope.go:117] "RemoveContainer" containerID="93c82489c9f35d1297898d2414a169ae3edfee51ac5229f02d7d2eee57aaaecf" Jul 22 16:08:03.217561 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:03.217335 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-sf2mn_openshift-console-operator(799e2b87-1954-4103-860d-33ab42170349)\"" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" podUID="799e2b87-1954-4103-860d-33ab42170349" Jul 22 16:08:04.408559 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:04.408530 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xspks_d0d50a09-e266-4478-8e93-19b3aea03d79/node-ca/0.log" Jul 22 16:08:04.948130 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:04.948095 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:04.948301 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:04.948201 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:04.948301 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:04.948220 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:08:04.948301 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:04.948282 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm podName:51215a2f-bce7-4a3d-a206-d96885c4c724 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:08.948266582 +0000 UTC m=+54.090140637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bbfb" (UID: "51215a2f-bce7-4a3d-a206-d96885c4c724") : configmap "kube-rbac-proxy" not found Jul 22 16:08:04.948407 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:04.948313 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:08:04.948407 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:04.948360 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls podName:51215a2f-bce7-4a3d-a206-d96885c4c724 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:08.948349674 +0000 UTC m=+54.090223729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bbfb" (UID: "51215a2f-bce7-4a3d-a206-d96885c4c724") : secret "insights-runtime-extractor-tls" not found Jul 22 16:08:05.015281 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:05.015254 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vprx2_bec7900e-a6b5-4e09-b908-690e4d13b982/serve-healthcheck-canary/0.log" Jul 22 16:08:07.066761 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:07.066720 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:08:07.067241 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:07.066907 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:08:07.067241 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:07.066933 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b465bb546-ppslg: secret "image-registry-tls" not found Jul 22 16:08:07.067241 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:07.067003 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls podName:1fe0626b-4d62-4845-b99c-9ff8d9e711e5 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:23.066982267 +0000 UTC m=+68.208856323 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls") pod "image-registry-6b465bb546-ppslg" (UID: "1fe0626b-4d62-4845-b99c-9ff8d9e711e5") : secret "image-registry-tls" not found Jul 22 16:08:07.168011 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:07.167977 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:08:07.168207 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:07.168052 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:08:07.168207 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:07.168083 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:08:07.168207 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:07.168119 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks2q4\" (UID: \"7e88090f-ac10-4518-954c-aa515ceb5cb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" Jul 22 16:08:07.168207 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:07.168146 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:08:07.168207 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:07.168146 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:08:07.168207 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:07.168201 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:08:07.168435 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:07.168237 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 16:08:07.168435 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:07.168258 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:08:07.168435 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:07.168241 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle podName:57dbd4ff-f4ed-484a-801b-9cef8a961409 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:23.168221191 +0000 UTC m=+68.310095264 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle") pod "router-default-586bddb798-57m2m" (UID: "57dbd4ff-f4ed-484a-801b-9cef8a961409") : configmap references non-existent config key: service-ca.crt Jul 22 16:08:07.168435 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:07.168310 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls podName:fa7a9636-f842-48f7-ae45-dfcca3c01ee3 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:23.168293823 +0000 UTC m=+68.310167882 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rg4vq" (UID: "fa7a9636-f842-48f7-ae45-dfcca3c01ee3") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:08:07.168435 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:07.168327 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls podName:db5fa8ab-0e89-4823-9f94-353bdf81e0dc nodeName:}" failed. No retries permitted until 2026-07-22 16:08:23.168319783 +0000 UTC m=+68.310193838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls") pod "dns-default-c78v9" (UID: "db5fa8ab-0e89-4823-9f94-353bdf81e0dc") : secret "dns-default-metrics-tls" not found Jul 22 16:08:07.168435 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:07.168336 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls podName:7e88090f-ac10-4518-954c-aa515ceb5cb7 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:23.1683315 +0000 UTC m=+68.310205555 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ks2q4" (UID: "7e88090f-ac10-4518-954c-aa515ceb5cb7") : secret "samples-operator-tls" not found Jul 22 16:08:07.168435 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:07.168346 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs podName:57dbd4ff-f4ed-484a-801b-9cef8a961409 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:23.168341294 +0000 UTC m=+68.310215349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs") pod "router-default-586bddb798-57m2m" (UID: "57dbd4ff-f4ed-484a-801b-9cef8a961409") : secret "router-metrics-certs-default" not found Jul 22 16:08:08.984536 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:08.984495 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:08.984998 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:08.984632 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:08.984998 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:08.984776 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:08:08.984998 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:08.984849 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls podName:51215a2f-bce7-4a3d-a206-d96885c4c724 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:16.984826219 +0000 UTC m=+62.126700277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bbfb" (UID: "51215a2f-bce7-4a3d-a206-d96885c4c724") : secret "insights-runtime-extractor-tls" not found Jul 22 16:08:08.985154 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:08.985134 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51215a2f-bce7-4a3d-a206-d96885c4c724-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:13.978843 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:13.978808 2567 scope.go:117] "RemoveContainer" containerID="93c82489c9f35d1297898d2414a169ae3edfee51ac5229f02d7d2eee57aaaecf" Jul 22 16:08:14.256721 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:14.256637 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:08:14.257084 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:14.257070 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/1.log" Jul 22 16:08:14.257135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:14.257100 2567 generic.go:356] "Generic (PLEG): container finished" podID="799e2b87-1954-4103-860d-33ab42170349" containerID="f399f2ac5b0452a90ef782f2a358d615c92c9b73fe9dfd631783e97be5eaedc5" exitCode=255 Jul 22 16:08:14.257173 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:14.257130 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" event={"ID":"799e2b87-1954-4103-860d-33ab42170349","Type":"ContainerDied","Data":"f399f2ac5b0452a90ef782f2a358d615c92c9b73fe9dfd631783e97be5eaedc5"} Jul 22 16:08:14.257215 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:14.257174 2567 scope.go:117] "RemoveContainer" containerID="93c82489c9f35d1297898d2414a169ae3edfee51ac5229f02d7d2eee57aaaecf" Jul 22 16:08:14.257518 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:14.257494 2567 scope.go:117] "RemoveContainer" containerID="f399f2ac5b0452a90ef782f2a358d615c92c9b73fe9dfd631783e97be5eaedc5" Jul 22 16:08:14.257718 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:14.257677 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-sf2mn_openshift-console-operator(799e2b87-1954-4103-860d-33ab42170349)\"" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" podUID="799e2b87-1954-4103-860d-33ab42170349" Jul 22 16:08:15.262031 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:15.262002 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:08:16.128829 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:16.128794 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g2pjb" Jul 22 16:08:17.056625 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:17.056564 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:17.058837 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:17.058812 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51215a2f-bce7-4a3d-a206-d96885c4c724-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bbfb\" (UID: \"51215a2f-bce7-4a3d-a206-d96885c4c724\") " pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:17.136543 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:17.136509 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cvcbr\"" Jul 22 16:08:17.143170 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:17.143137 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5bbfb" Jul 22 16:08:17.291086 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:17.291057 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5bbfb"] Jul 22 16:08:17.296497 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:08:17.296471 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51215a2f_bce7_4a3d_a206_d96885c4c724.slice/crio-5701fa1f8b91e64ba9a2d1753fe88bbda698ed38626d1cac82fc56c44ba0d925 WatchSource:0}: Error finding container 5701fa1f8b91e64ba9a2d1753fe88bbda698ed38626d1cac82fc56c44ba0d925: Status 404 returned error can't find the container with id 5701fa1f8b91e64ba9a2d1753fe88bbda698ed38626d1cac82fc56c44ba0d925 Jul 22 16:08:18.273006 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:18.272973 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bbfb" event={"ID":"51215a2f-bce7-4a3d-a206-d96885c4c724","Type":"ContainerStarted","Data":"5701fa1f8b91e64ba9a2d1753fe88bbda698ed38626d1cac82fc56c44ba0d925"} Jul 22 16:08:21.499468 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.499421 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs\") pod \"network-metrics-daemon-6vr2t\" (UID: \"8c507486-5095-4e26-ba26-9ac77c17590d\") " pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:08:21.499966 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.499505 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:08:21.505897 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.505865 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 16:08:21.507157 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.507135 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 16:08:21.512456 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.512432 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c507486-5095-4e26-ba26-9ac77c17590d-metrics-certs\") pod \"network-metrics-daemon-6vr2t\" (UID: \"8c507486-5095-4e26-ba26-9ac77c17590d\") " pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:08:21.512567 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.512492 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a19af359-cd18-4594-817d-a115665176bd-original-pull-secret\") pod \"global-pull-secret-syncer-f7rz8\" (UID: \"a19af359-cd18-4594-817d-a115665176bd\") " pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:08:21.592881 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.592841 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:08:21.593048 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.592927 2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:08:21.593291 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.593274 2567 scope.go:117] "RemoveContainer" containerID="f399f2ac5b0452a90ef782f2a358d615c92c9b73fe9dfd631783e97be5eaedc5" Jul 22 16:08:21.593477 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:21.593455 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-sf2mn_openshift-console-operator(799e2b87-1954-4103-860d-33ab42170349)\"" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" podUID="799e2b87-1954-4103-860d-33ab42170349" Jul 22 16:08:21.600371 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.600349 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs46p\" (UniqueName: \"kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p\") pod \"network-check-target-rnnbk\" (UID: \"fb688b7c-f404-456b-a23e-933a5c2c7fae\") " pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:08:21.603040 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.602989 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs46p\" (UniqueName: \"kubernetes.io/projected/fb688b7c-f404-456b-a23e-933a5c2c7fae-kube-api-access-xs46p\") pod \"network-check-target-rnnbk\" (UID: \"fb688b7c-f404-456b-a23e-933a5c2c7fae\") " pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:08:21.690647 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.690609 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f7rz8" Jul 22 16:08:21.699427 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.699403 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rb5n6\"" Jul 22 16:08:21.704558 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.704535 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s68m6\"" Jul 22 16:08:21.706291 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.706267 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vr2t" Jul 22 16:08:21.711003 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:21.710985 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:08:22.124503 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:22.124329 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rnnbk"] Jul 22 16:08:22.127929 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:08:22.127896 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb688b7c_f404_456b_a23e_933a5c2c7fae.slice/crio-fe7f8857c4187fb9e6a2f723c2e81dc96fac3914cfd84226260098ce94592b8c WatchSource:0}: Error finding container fe7f8857c4187fb9e6a2f723c2e81dc96fac3914cfd84226260098ce94592b8c: Status 404 returned error can't find the container with id fe7f8857c4187fb9e6a2f723c2e81dc96fac3914cfd84226260098ce94592b8c Jul 22 16:08:22.148447 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:22.148419 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f7rz8"] Jul 22 16:08:22.157985 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:08:22.157952 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19af359_cd18_4594_817d_a115665176bd.slice/crio-0946dbebd20a275234026213e6c8e36fc7f59db0e1170f6fcc6a527b21edf1ec WatchSource:0}: Error finding container 0946dbebd20a275234026213e6c8e36fc7f59db0e1170f6fcc6a527b21edf1ec: Status 404 returned error can't find the container with id 0946dbebd20a275234026213e6c8e36fc7f59db0e1170f6fcc6a527b21edf1ec Jul 22 16:08:22.166420 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:22.166358 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6vr2t"] Jul 22 16:08:22.176720 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:08:22.176683 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c507486_5095_4e26_ba26_9ac77c17590d.slice/crio-99b561ff6975eee9cb06207dfd407782155c863cb40e67b97b80009aa3a0f3c5 WatchSource:0}: Error finding container 99b561ff6975eee9cb06207dfd407782155c863cb40e67b97b80009aa3a0f3c5: Status 404 returned error can't find the container with id 99b561ff6975eee9cb06207dfd407782155c863cb40e67b97b80009aa3a0f3c5 Jul 22 16:08:22.286528 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:22.286494 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6vr2t" event={"ID":"8c507486-5095-4e26-ba26-9ac77c17590d","Type":"ContainerStarted","Data":"99b561ff6975eee9cb06207dfd407782155c863cb40e67b97b80009aa3a0f3c5"} Jul 22 16:08:22.287625 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:22.287593 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f7rz8" event={"ID":"a19af359-cd18-4594-817d-a115665176bd","Type":"ContainerStarted","Data":"0946dbebd20a275234026213e6c8e36fc7f59db0e1170f6fcc6a527b21edf1ec"} Jul 22 16:08:22.288938 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:22.288916 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bbfb" event={"ID":"51215a2f-bce7-4a3d-a206-d96885c4c724","Type":"ContainerStarted","Data":"036d792003b761820f3bb52881184cab6d916e4148b5cd994d922a62f7ac343d"} Jul 22 16:08:22.290201 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:22.290173 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rnnbk" event={"ID":"fb688b7c-f404-456b-a23e-933a5c2c7fae","Type":"ContainerStarted","Data":"0dba5e83ba5c6af4be57a3841097c5c7b29d5deabc6206aaa9e3cc0dabbff6f0"} Jul 22 16:08:22.290281 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:22.290208 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rnnbk" event={"ID":"fb688b7c-f404-456b-a23e-933a5c2c7fae","Type":"ContainerStarted","Data":"fe7f8857c4187fb9e6a2f723c2e81dc96fac3914cfd84226260098ce94592b8c"} Jul 22 16:08:22.290315 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:22.290307 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:08:22.308956 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:22.308916 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rnnbk" podStartSLOduration=66.308903084 podStartE2EDuration="1m6.308903084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:22.308709515 +0000 UTC m=+67.450583596" watchObservedRunningTime="2026-07-22 16:08:22.308903084 +0000 UTC m=+67.450777154" Jul 22 16:08:23.117036 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.116842 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:08:23.120130 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.119344 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls\") pod \"image-registry-6b465bb546-ppslg\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:08:23.218196 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.218151 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks2q4\" (UID: \"7e88090f-ac10-4518-954c-aa515ceb5cb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" Jul 22 16:08:23.218388 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.218217 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:08:23.218388 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.218270 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:08:23.218388 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.218334 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:08:23.218388 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.218374 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:08:23.219276 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.219221 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57dbd4ff-f4ed-484a-801b-9cef8a961409-service-ca-bundle\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:08:23.221079 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.221031 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db5fa8ab-0e89-4823-9f94-353bdf81e0dc-metrics-tls\") pod \"dns-default-c78v9\" (UID: \"db5fa8ab-0e89-4823-9f94-353bdf81e0dc\") " pod="openshift-dns/dns-default-c78v9" Jul 22 16:08:23.221191 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.221174 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e88090f-ac10-4518-954c-aa515ceb5cb7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks2q4\" (UID: \"7e88090f-ac10-4518-954c-aa515ceb5cb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" Jul 22 16:08:23.221461 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.221437 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57dbd4ff-f4ed-484a-801b-9cef8a961409-metrics-certs\") pod \"router-default-586bddb798-57m2m\" (UID: \"57dbd4ff-f4ed-484a-801b-9cef8a961409\") " pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:08:23.221568 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.221481 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa7a9636-f842-48f7-ae45-dfcca3c01ee3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rg4vq\" (UID: \"fa7a9636-f842-48f7-ae45-dfcca3c01ee3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:08:23.305489 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.305459 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-crmjn\"" Jul 22 16:08:23.312982 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.312952 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:08:23.334226 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.334201 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wdbjp\"" Jul 22 16:08:23.340108 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.339922 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vmv54\"" Jul 22 16:08:23.341395 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.341152 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c78v9" Jul 22 16:08:23.347882 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.347857 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:08:23.376336 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.376287 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-28lg6\"" Jul 22 16:08:23.380303 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.380139 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gljqq\"" Jul 22 16:08:23.384268 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.384186 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" Jul 22 16:08:23.387790 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.387760 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" Jul 22 16:08:23.504051 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.504017 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b465bb546-ppslg"] Jul 22 16:08:23.548271 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.548151 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c78v9"] Jul 22 16:08:23.782454 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:08:23.782418 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5fa8ab_0e89_4823_9f94_353bdf81e0dc.slice/crio-c3a4c8bb47d99aaf842fd3b84833eb578c39db4b5e56ede182dce829fe491f2a WatchSource:0}: Error finding container c3a4c8bb47d99aaf842fd3b84833eb578c39db4b5e56ede182dce829fe491f2a: Status 404 returned error can't find the container with id c3a4c8bb47d99aaf842fd3b84833eb578c39db4b5e56ede182dce829fe491f2a Jul 22 16:08:23.783823 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:08:23.783780 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe0626b_4d62_4845_b99c_9ff8d9e711e5.slice/crio-40bb0dddcfb473c83f2ca5691c2cadc1592fd0e028afa9df4f2b1fb125a2f5cd WatchSource:0}: Error finding container 40bb0dddcfb473c83f2ca5691c2cadc1592fd0e028afa9df4f2b1fb125a2f5cd: Status 404 returned error can't find the container with id 40bb0dddcfb473c83f2ca5691c2cadc1592fd0e028afa9df4f2b1fb125a2f5cd Jul 22 16:08:23.969023 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:23.968988 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq"] Jul 22 16:08:24.077324 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:08:24.077251 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7a9636_f842_48f7_ae45_dfcca3c01ee3.slice/crio-490bef2b279873adca6341ceb6fbc370ef87f6b764b98d06133b99d1a23e683b WatchSource:0}: Error finding container 490bef2b279873adca6341ceb6fbc370ef87f6b764b98d06133b99d1a23e683b: Status 404 returned error can't find the container with id 490bef2b279873adca6341ceb6fbc370ef87f6b764b98d06133b99d1a23e683b Jul 22 16:08:24.217493 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:24.217396 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4"] Jul 22 16:08:24.223945 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:24.223911 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-586bddb798-57m2m"] Jul 22 16:08:24.299083 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:24.299047 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c78v9" event={"ID":"db5fa8ab-0e89-4823-9f94-353bdf81e0dc","Type":"ContainerStarted","Data":"c3a4c8bb47d99aaf842fd3b84833eb578c39db4b5e56ede182dce829fe491f2a"} Jul 22 16:08:24.300440 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:24.300381 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" event={"ID":"fa7a9636-f842-48f7-ae45-dfcca3c01ee3","Type":"ContainerStarted","Data":"490bef2b279873adca6341ceb6fbc370ef87f6b764b98d06133b99d1a23e683b"} Jul 22 16:08:24.302127 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:24.302098 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b465bb546-ppslg" event={"ID":"1fe0626b-4d62-4845-b99c-9ff8d9e711e5","Type":"ContainerStarted","Data":"8b085b7c8b045bb347efe43f09c05c99a1e942a29c50e57e1c84c1c26408d0c1"} Jul 22 16:08:24.302241 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:24.302145 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b465bb546-ppslg" event={"ID":"1fe0626b-4d62-4845-b99c-9ff8d9e711e5","Type":"ContainerStarted","Data":"40bb0dddcfb473c83f2ca5691c2cadc1592fd0e028afa9df4f2b1fb125a2f5cd"} Jul 22 16:08:24.302563 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:24.302522 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:08:26.314532 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:26.314489 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" event={"ID":"7e88090f-ac10-4518-954c-aa515ceb5cb7","Type":"ContainerStarted","Data":"c0448dddb6202fdab1a871e112408fe33a5d226c9871da497ecd3b64c585e7e8"} Jul 22 16:08:27.468663 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:08:27.468628 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57dbd4ff_f4ed_484a_801b_9cef8a961409.slice/crio-db44f556962464095895b2f828e0a257f7e30b29f88c85b4de01df31cfe969bc WatchSource:0}: Error finding container db44f556962464095895b2f828e0a257f7e30b29f88c85b4de01df31cfe969bc: Status 404 returned error can't find the container with id db44f556962464095895b2f828e0a257f7e30b29f88c85b4de01df31cfe969bc Jul 22 16:08:28.326512 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:28.326441 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-586bddb798-57m2m" event={"ID":"57dbd4ff-f4ed-484a-801b-9cef8a961409","Type":"ContainerStarted","Data":"db44f556962464095895b2f828e0a257f7e30b29f88c85b4de01df31cfe969bc"} Jul 22 16:08:29.331446 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:29.331406 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6vr2t" event={"ID":"8c507486-5095-4e26-ba26-9ac77c17590d","Type":"ContainerStarted","Data":"a5b66a61cd5f09ca3000cc3dbc4b2388744082a9b03b58474151319306fe938b"} Jul 22 16:08:29.332767 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:29.332734 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f7rz8" event={"ID":"a19af359-cd18-4594-817d-a115665176bd","Type":"ContainerStarted","Data":"97a7b911693f930404b9f853d94ff5de87e8ed86c103aabc5c52063246582184"} Jul 22 16:08:29.334129 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:29.334106 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-586bddb798-57m2m" event={"ID":"57dbd4ff-f4ed-484a-801b-9cef8a961409","Type":"ContainerStarted","Data":"9fe872a9b474564be38b206b83849efe32adfb54afe6572ea67ff6dc8cba6313"} Jul 22 16:08:29.348171 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:29.348143 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:08:29.350830 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:29.350809 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:08:29.354749 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:29.354712 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b465bb546-ppslg" podStartSLOduration=73.354678137 podStartE2EDuration="1m13.354678137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:24.33507301 +0000 UTC m=+69.476947112" watchObservedRunningTime="2026-07-22 16:08:29.354678137 +0000 UTC m=+74.496552213" Jul 22 16:08:29.355463 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:29.355430 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f7rz8" podStartSLOduration=67.012578557 podStartE2EDuration="1m13.355421977s" totalImagesPullingTime="6.34284342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:29.353843354 +0000 UTC m=+74.495717443" watchObservedRunningTime="2026-07-22 16:08:29.355421977 +0000 UTC m=+74.497296053" Jul 22 16:08:29.395397 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:29.395345 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-586bddb798-57m2m" podStartSLOduration=59.395329612 podStartE2EDuration="59.395329612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:29.393680566 +0000 UTC m=+74.535554643" watchObservedRunningTime="2026-07-22 16:08:29.395329612 +0000 UTC m=+74.537203688" Jul 22 16:08:30.338775 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:30.338734 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:08:30.340045 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:30.340020 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-586bddb798-57m2m" Jul 22 16:08:31.344327 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:31.344297 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" event={"ID":"7e88090f-ac10-4518-954c-aa515ceb5cb7","Type":"ContainerStarted","Data":"df4fa3fc7ee1982c5338b23affa5da46c4c0a854fa4ceaba241ed2e55439dcf2"} Jul 22 16:08:31.346502 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:31.346474 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bbfb" event={"ID":"51215a2f-bce7-4a3d-a206-d96885c4c724","Type":"ContainerStarted","Data":"a1f5f83af8309cfeae61d37a40f300a14e875605f6acde14bc06be37298df336"} Jul 22 16:08:31.347831 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:31.347803 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c78v9" event={"ID":"db5fa8ab-0e89-4823-9f94-353bdf81e0dc","Type":"ContainerStarted","Data":"5824aee0aee9e15a420856fb8fb91247b6cca82125c4e1c1a71eaddddd7078d9"} Jul 22 16:08:31.351336 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:31.351306 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6vr2t" event={"ID":"8c507486-5095-4e26-ba26-9ac77c17590d","Type":"ContainerStarted","Data":"822e1e8415a859635bab0edcc9bf0ae27200b9568866f6812fab815ef6e4bc7d"} Jul 22 16:08:31.355254 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:31.354157 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" event={"ID":"fa7a9636-f842-48f7-ae45-dfcca3c01ee3","Type":"ContainerStarted","Data":"bf8d08b8b1efc7ebb859c1bd2fb17a1f66ea87cdc8550a1e2c1f8ef1323f680f"} Jul 22 16:08:31.376911 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:31.376856 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6vr2t" podStartSLOduration=69.09191197 podStartE2EDuration="1m15.376838191s" totalImagesPullingTime="6.284926221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:31.375558523 +0000 UTC m=+76.517432599" watchObservedRunningTime="2026-07-22 16:08:31.376838191 +0000 UTC m=+76.518712267" Jul 22 16:08:31.394223 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:31.394176 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rg4vq" podStartSLOduration=54.422437689 podStartE2EDuration="1m1.394159456s" totalImagesPullingTime="6.971721767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:31.393648474 +0000 UTC m=+76.535522574" watchObservedRunningTime="2026-07-22 16:08:31.394159456 +0000 UTC m=+76.536033532" Jul 22 16:08:31.978341 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:31.978305 2567 scope.go:117] "RemoveContainer" containerID="f399f2ac5b0452a90ef782f2a358d615c92c9b73fe9dfd631783e97be5eaedc5" Jul 22 16:08:31.978526 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:31.978499 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-sf2mn_openshift-console-operator(799e2b87-1954-4103-860d-33ab42170349)\"" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" podUID="799e2b87-1954-4103-860d-33ab42170349" Jul 22 16:08:32.360293 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:32.360253 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c78v9" event={"ID":"db5fa8ab-0e89-4823-9f94-353bdf81e0dc","Type":"ContainerStarted","Data":"be53cf35f24684d3f0bf3e1a5beaa1b81d3f4527366510df2b96e512f37c24ed"} Jul 22 16:08:32.360749 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:32.360420 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c78v9" Jul 22 16:08:32.362170 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:32.362131 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" event={"ID":"7e88090f-ac10-4518-954c-aa515ceb5cb7","Type":"ContainerStarted","Data":"f40f11ea97b19c3d6401c2aca80d4a213c08503ec6c4f300ae615f3bbc0a0d69"} Jul 22 16:08:32.381582 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:32.381528 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c78v9" podStartSLOduration=34.116352853 podStartE2EDuration="41.381511035s" totalImagesPullingTime="7.265158182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:32.380535125 +0000 UTC m=+77.522409231" watchObservedRunningTime="2026-07-22 16:08:32.381511035 +0000 UTC m=+77.523385112" Jul 22 16:08:32.399309 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:32.399245 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks2q4" podStartSLOduration=48.208787341 podStartE2EDuration="53.399224211s" totalImagesPullingTime="5.19043687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:32.397659195 +0000 UTC m=+77.539533292" watchObservedRunningTime="2026-07-22 16:08:32.399224211 +0000 UTC m=+77.541098289" Jul 22 16:08:35.376104 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:35.376066 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bbfb" event={"ID":"51215a2f-bce7-4a3d-a206-d96885c4c724","Type":"ContainerStarted","Data":"da12b89672f443896c37113f7f90bd47ca3cd5bd340e077e4e5e8561a17ade1b"} Jul 22 16:08:35.395084 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:35.395020 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5bbfb" podStartSLOduration=17.395289582 podStartE2EDuration="34.395003244s" totalImagesPullingTime="16.999713662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:08:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:35.393846912 +0000 UTC m=+80.535721012" watchObservedRunningTime="2026-07-22 16:08:35.395003244 +0000 UTC m=+80.536877320" Jul 22 16:08:37.759896 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.759767 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-h26ps"] Jul 22 16:08:37.762071 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.762054 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" Jul 22 16:08:37.764700 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.764672 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 22 16:08:37.764821 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.764672 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 22 16:08:37.765511 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.765486 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-v8hc8\"" Jul 22 16:08:37.765597 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.765578 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 16:08:37.774363 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.774343 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-h26ps"] Jul 22 16:08:37.833813 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.833768 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/178d03dd-d01a-40a2-a753-9cb23e70d8d1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-h26ps\" (UID: \"178d03dd-d01a-40a2-a753-9cb23e70d8d1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" Jul 22 16:08:37.833813 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.833811 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g857f\" (UniqueName: \"kubernetes.io/projected/178d03dd-d01a-40a2-a753-9cb23e70d8d1-kube-api-access-g857f\") pod \"prometheus-operator-5945b5cfd-h26ps\" (UID: \"178d03dd-d01a-40a2-a753-9cb23e70d8d1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" Jul 22 16:08:37.834013 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.833849 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/178d03dd-d01a-40a2-a753-9cb23e70d8d1-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-h26ps\" (UID: \"178d03dd-d01a-40a2-a753-9cb23e70d8d1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" Jul 22 16:08:37.834013 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.833921 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/178d03dd-d01a-40a2-a753-9cb23e70d8d1-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-h26ps\" (UID: \"178d03dd-d01a-40a2-a753-9cb23e70d8d1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" Jul 22 16:08:37.934983 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.934938 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/178d03dd-d01a-40a2-a753-9cb23e70d8d1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-h26ps\" (UID: \"178d03dd-d01a-40a2-a753-9cb23e70d8d1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" Jul 22 16:08:37.934983 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.934980 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g857f\" (UniqueName: \"kubernetes.io/projected/178d03dd-d01a-40a2-a753-9cb23e70d8d1-kube-api-access-g857f\") pod \"prometheus-operator-5945b5cfd-h26ps\" (UID: \"178d03dd-d01a-40a2-a753-9cb23e70d8d1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" Jul 22 16:08:37.935196 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.935052 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/178d03dd-d01a-40a2-a753-9cb23e70d8d1-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-h26ps\" (UID: \"178d03dd-d01a-40a2-a753-9cb23e70d8d1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" Jul 22 16:08:37.935196 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.935074 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/178d03dd-d01a-40a2-a753-9cb23e70d8d1-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-h26ps\" (UID: \"178d03dd-d01a-40a2-a753-9cb23e70d8d1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" Jul 22 16:08:37.935718 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.935671 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/178d03dd-d01a-40a2-a753-9cb23e70d8d1-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-h26ps\" (UID: \"178d03dd-d01a-40a2-a753-9cb23e70d8d1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" Jul 22 16:08:37.937422 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.937399 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/178d03dd-d01a-40a2-a753-9cb23e70d8d1-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-h26ps\" (UID: \"178d03dd-d01a-40a2-a753-9cb23e70d8d1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" Jul 22 16:08:37.937520 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.937401 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/178d03dd-d01a-40a2-a753-9cb23e70d8d1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-h26ps\" (UID: \"178d03dd-d01a-40a2-a753-9cb23e70d8d1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" Jul 22 16:08:37.944461 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:37.944426 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g857f\" (UniqueName: \"kubernetes.io/projected/178d03dd-d01a-40a2-a753-9cb23e70d8d1-kube-api-access-g857f\") pod \"prometheus-operator-5945b5cfd-h26ps\" (UID: \"178d03dd-d01a-40a2-a753-9cb23e70d8d1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" Jul 22 16:08:38.070423 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:38.070378 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" Jul 22 16:08:38.200635 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:38.200593 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-h26ps"] Jul 22 16:08:38.204281 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:08:38.204255 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178d03dd_d01a_40a2_a753_9cb23e70d8d1.slice/crio-c09fcf1f6cad29727ab5e5a3d5a05f833ebd13e33f1e7390e6cbfdae2b7047af WatchSource:0}: Error finding container c09fcf1f6cad29727ab5e5a3d5a05f833ebd13e33f1e7390e6cbfdae2b7047af: Status 404 returned error can't find the container with id c09fcf1f6cad29727ab5e5a3d5a05f833ebd13e33f1e7390e6cbfdae2b7047af Jul 22 16:08:38.387655 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:38.387573 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" event={"ID":"178d03dd-d01a-40a2-a753-9cb23e70d8d1","Type":"ContainerStarted","Data":"c09fcf1f6cad29727ab5e5a3d5a05f833ebd13e33f1e7390e6cbfdae2b7047af"} Jul 22 16:08:40.396372 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:40.396334 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" event={"ID":"178d03dd-d01a-40a2-a753-9cb23e70d8d1","Type":"ContainerStarted","Data":"b2a51ceaaf2fb6b57423456237cfb73e732a7380ea33bcd522e86c0ce4ba2f0a"} Jul 22 16:08:40.396766 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:40.396389 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" event={"ID":"178d03dd-d01a-40a2-a753-9cb23e70d8d1","Type":"ContainerStarted","Data":"86ce774ee61e5d2114abcb6594852ec47ee772b294212ded6b6c30811bb86e1c"} Jul 22 16:08:40.415617 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:40.415563 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-h26ps" podStartSLOduration=2.251845033 podStartE2EDuration="3.415546189s" totalImagesPullingTime="1.163701156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:40.414445562 +0000 UTC m=+85.556319651" watchObservedRunningTime="2026-07-22 16:08:40.415546189 +0000 UTC m=+85.557420265" Jul 22 16:08:42.135559 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.135517 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-82pl8"] Jul 22 16:08:42.139140 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.139119 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.142569 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.142542 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 22 16:08:42.142679 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.142547 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 22 16:08:42.142679 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.142576 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v6gxz\"" Jul 22 16:08:42.143368 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.143352 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 22 16:08:42.146200 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.146176 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8p79m"] Jul 22 16:08:42.149402 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.149381 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.152032 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.152009 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 16:08:42.152032 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.152035 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 16:08:42.152312 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.152293 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 16:08:42.152442 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.152424 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jqfxs\"" Jul 22 16:08:42.153740 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.153720 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-82pl8"] Jul 22 16:08:42.168674 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.168636 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e956f24d-9f22-4042-977e-e455a073722a-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.168674 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.168671 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e956f24d-9f22-4042-977e-e455a073722a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.168908 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.168715 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e956f24d-9f22-4042-977e-e455a073722a-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.168908 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.168803 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dce4f379-8057-4575-afa2-08b971bf9c79-sys\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.168908 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.168835 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dce4f379-8057-4575-afa2-08b971bf9c79-metrics-client-ca\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.169007 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.168920 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e956f24d-9f22-4042-977e-e455a073722a-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.169007 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.168944 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-accelerators-collector-config\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.169007 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.168970 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-wtmp\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.169007 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.169002 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-textfile\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.169135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.169028 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dce4f379-8057-4575-afa2-08b971bf9c79-root\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.169135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.169045 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-tls\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.169135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.169067 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e956f24d-9f22-4042-977e-e455a073722a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.169135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.169099 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.169135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.169123 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59ndd\" (UniqueName: \"kubernetes.io/projected/dce4f379-8057-4575-afa2-08b971bf9c79-kube-api-access-59ndd\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.169298 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.169161 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd8tc\" (UniqueName: \"kubernetes.io/projected/e956f24d-9f22-4042-977e-e455a073722a-kube-api-access-dd8tc\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.270242 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270200 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59ndd\" (UniqueName: \"kubernetes.io/projected/dce4f379-8057-4575-afa2-08b971bf9c79-kube-api-access-59ndd\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.270242 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270241 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd8tc\" (UniqueName: \"kubernetes.io/projected/e956f24d-9f22-4042-977e-e455a073722a-kube-api-access-dd8tc\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.270483 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270467 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e956f24d-9f22-4042-977e-e455a073722a-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.270518 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270493 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e956f24d-9f22-4042-977e-e455a073722a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.270551 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270527 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e956f24d-9f22-4042-977e-e455a073722a-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.270628 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270558 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dce4f379-8057-4575-afa2-08b971bf9c79-sys\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.270628 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270611 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dce4f379-8057-4575-afa2-08b971bf9c79-sys\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.270718 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270645 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dce4f379-8057-4575-afa2-08b971bf9c79-metrics-client-ca\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.270718 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270678 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e956f24d-9f22-4042-977e-e455a073722a-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.270813 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270729 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-accelerators-collector-config\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.270813 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270765 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-wtmp\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.270813 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270800 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-textfile\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.270953 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270839 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dce4f379-8057-4575-afa2-08b971bf9c79-root\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.270953 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270861 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-tls\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.270953 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270894 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e956f24d-9f22-4042-977e-e455a073722a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.270953 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270926 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.271143 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.270989 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e956f24d-9f22-4042-977e-e455a073722a-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.271143 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.271069 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dce4f379-8057-4575-afa2-08b971bf9c79-root\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.271240 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.271191 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-wtmp\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.271359 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.271321 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-accelerators-collector-config\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.271359 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.271321 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dce4f379-8057-4575-afa2-08b971bf9c79-metrics-client-ca\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.271516 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:42.271402 2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 16:08:42.271516 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:08:42.271459 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-tls podName:dce4f379-8057-4575-afa2-08b971bf9c79 nodeName:}" failed. No retries permitted until 2026-07-22 16:08:42.771440391 +0000 UTC m=+87.913314447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-tls") pod "node-exporter-8p79m" (UID: "dce4f379-8057-4575-afa2-08b971bf9c79") : secret "node-exporter-tls" not found Jul 22 16:08:42.271516 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.271472 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e956f24d-9f22-4042-977e-e455a073722a-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.271516 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.271499 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-textfile\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.272057 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.272038 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e956f24d-9f22-4042-977e-e455a073722a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.273092 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.273073 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e956f24d-9f22-4042-977e-e455a073722a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.273339 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.273320 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.273472 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.273454 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e956f24d-9f22-4042-977e-e455a073722a-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.278703 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.278667 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59ndd\" (UniqueName: \"kubernetes.io/projected/dce4f379-8057-4575-afa2-08b971bf9c79-kube-api-access-59ndd\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.278978 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.278949 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd8tc\" (UniqueName: \"kubernetes.io/projected/e956f24d-9f22-4042-977e-e455a073722a-kube-api-access-dd8tc\") pod \"kube-state-metrics-56c5594674-82pl8\" (UID: \"e956f24d-9f22-4042-977e-e455a073722a\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.368039 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.368012 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c78v9" Jul 22 16:08:42.448570 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.448496 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" Jul 22 16:08:42.580450 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.580413 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-82pl8"] Jul 22 16:08:42.583196 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:08:42.583164 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode956f24d_9f22_4042_977e_e455a073722a.slice/crio-853de9a37537ed3aeedd3ea4936e81bfc88b7d7ea931bc3fb222f1e1660096c6 WatchSource:0}: Error finding container 853de9a37537ed3aeedd3ea4936e81bfc88b7d7ea931bc3fb222f1e1660096c6: Status 404 returned error can't find the container with id 853de9a37537ed3aeedd3ea4936e81bfc88b7d7ea931bc3fb222f1e1660096c6 Jul 22 16:08:42.774054 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.774023 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-tls\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:42.776257 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:42.776227 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dce4f379-8057-4575-afa2-08b971bf9c79-node-exporter-tls\") pod \"node-exporter-8p79m\" (UID: \"dce4f379-8057-4575-afa2-08b971bf9c79\") " pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:43.058994 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:43.058927 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8p79m" Jul 22 16:08:43.070983 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:08:43.070954 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce4f379_8057_4575_afa2_08b971bf9c79.slice/crio-d4cbecc51d40de90a4353196348c4bc71633b69fb47afc3a5372d6d19f924aab WatchSource:0}: Error finding container d4cbecc51d40de90a4353196348c4bc71633b69fb47afc3a5372d6d19f924aab: Status 404 returned error can't find the container with id d4cbecc51d40de90a4353196348c4bc71633b69fb47afc3a5372d6d19f924aab Jul 22 16:08:43.318975 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:43.318877 2567 patch_prober.go:28] interesting pod/image-registry-6b465bb546-ppslg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 22 16:08:43.318975 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:43.318936 2567 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6b465bb546-ppslg" podUID="1fe0626b-4d62-4845-b99c-9ff8d9e711e5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 16:08:43.411228 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:43.411188 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8p79m" event={"ID":"dce4f379-8057-4575-afa2-08b971bf9c79","Type":"ContainerStarted","Data":"d4cbecc51d40de90a4353196348c4bc71633b69fb47afc3a5372d6d19f924aab"} Jul 22 16:08:43.412593 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:43.412554 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" event={"ID":"e956f24d-9f22-4042-977e-e455a073722a","Type":"ContainerStarted","Data":"853de9a37537ed3aeedd3ea4936e81bfc88b7d7ea931bc3fb222f1e1660096c6"} Jul 22 16:08:44.420371 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:44.420058 2567 generic.go:356] "Generic (PLEG): container finished" podID="dce4f379-8057-4575-afa2-08b971bf9c79" containerID="0a980fce4b50120b2e28465c733ae66f569de0e03ccac563fe22bde9a4a69709" exitCode=0 Jul 22 16:08:44.420371 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:44.420145 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8p79m" event={"ID":"dce4f379-8057-4575-afa2-08b971bf9c79","Type":"ContainerDied","Data":"0a980fce4b50120b2e28465c733ae66f569de0e03ccac563fe22bde9a4a69709"} Jul 22 16:08:44.423417 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:44.423378 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" event={"ID":"e956f24d-9f22-4042-977e-e455a073722a","Type":"ContainerStarted","Data":"5a4f931e8212cbee8fb4452a5df88ad4cf432f24c7ffd995de268bab1dcda337"} Jul 22 16:08:44.423566 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:44.423451 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" event={"ID":"e956f24d-9f22-4042-977e-e455a073722a","Type":"ContainerStarted","Data":"2213ea9060265ac5a05b2180b3c4d04184291755519287957c967c8b509aa503"} Jul 22 16:08:45.313280 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:45.313252 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:08:45.429970 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:45.429929 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8p79m" event={"ID":"dce4f379-8057-4575-afa2-08b971bf9c79","Type":"ContainerStarted","Data":"78aa0b78b18087fea96bea9f54c96d33074439836be0f778e5d95333098dde53"} Jul 22 16:08:45.430403 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:45.429980 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8p79m" event={"ID":"dce4f379-8057-4575-afa2-08b971bf9c79","Type":"ContainerStarted","Data":"77453a84255d93d0fc6c6236e57a1577b4e31155b35e75b27918dc2f7666e169"} Jul 22 16:08:45.432085 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:45.432054 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" event={"ID":"e956f24d-9f22-4042-977e-e455a073722a","Type":"ContainerStarted","Data":"32cca625275a504ca2d399adcc340f12162bb6d043b9e8f8ea3d58ab29776964"} Jul 22 16:08:45.455343 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:45.455289 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8p79m" podStartSLOduration=2.246382335 podStartE2EDuration="3.455272748s" totalImagesPullingTime="1.117825119s" totalInitContainerRuntime="91.065294ms" isStatefulPod=true podCreationTimestamp="2026-07-22 16:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:45.453933577 +0000 UTC m=+90.595807679" watchObservedRunningTime="2026-07-22 16:08:45.455272748 +0000 UTC m=+90.597146825" Jul 22 16:08:45.473728 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:45.473661 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-82pl8" podStartSLOduration=1.868758749 podStartE2EDuration="3.47364449s" totalImagesPullingTime="1.604885741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:45.472504312 +0000 UTC m=+90.614378389" watchObservedRunningTime="2026-07-22 16:08:45.47364449 +0000 UTC m=+90.615518579" Jul 22 16:08:46.490450 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.490414 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6b8df8fc77-n9zx4"] Jul 22 16:08:46.493757 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.493739 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.498816 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.498765 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d802afjjh3erg\"" Jul 22 16:08:46.498816 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.498766 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 22 16:08:46.499123 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.499102 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 22 16:08:46.499242 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.499146 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 22 16:08:46.499242 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.499173 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 16:08:46.499242 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.499178 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rxwnv\"" Jul 22 16:08:46.520511 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.520480 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b8df8fc77-n9zx4"] Jul 22 16:08:46.606463 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.606426 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2f07ce17-c736-4e6f-bd59-f6c84de3b161-secret-metrics-server-client-certs\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.606463 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.606469 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2f07ce17-c736-4e6f-bd59-f6c84de3b161-audit-log\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.606682 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.606499 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f07ce17-c736-4e6f-bd59-f6c84de3b161-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.606682 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.606549 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2f07ce17-c736-4e6f-bd59-f6c84de3b161-secret-metrics-server-tls\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.606682 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.606639 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dxkl\" (UniqueName: \"kubernetes.io/projected/2f07ce17-c736-4e6f-bd59-f6c84de3b161-kube-api-access-5dxkl\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.606809 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.606683 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2f07ce17-c736-4e6f-bd59-f6c84de3b161-metrics-server-audit-profiles\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.606809 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.606789 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2f07ce17-c736-4e6f-bd59-f6c84de3b161-client-ca-bundle\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.707671 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.707632 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dxkl\" (UniqueName: \"kubernetes.io/projected/2f07ce17-c736-4e6f-bd59-f6c84de3b161-kube-api-access-5dxkl\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.707671 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.707673 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2f07ce17-c736-4e6f-bd59-f6c84de3b161-metrics-server-audit-profiles\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.707829 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.707715 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2f07ce17-c736-4e6f-bd59-f6c84de3b161-client-ca-bundle\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.707829 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.707755 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2f07ce17-c736-4e6f-bd59-f6c84de3b161-secret-metrics-server-client-certs\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.707829 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.707786 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2f07ce17-c736-4e6f-bd59-f6c84de3b161-audit-log\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.707829 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.707820 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f07ce17-c736-4e6f-bd59-f6c84de3b161-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.708023 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.707845 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2f07ce17-c736-4e6f-bd59-f6c84de3b161-secret-metrics-server-tls\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.708233 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.708208 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2f07ce17-c736-4e6f-bd59-f6c84de3b161-audit-log\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.708548 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.708526 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f07ce17-c736-4e6f-bd59-f6c84de3b161-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.708671 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.708652 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2f07ce17-c736-4e6f-bd59-f6c84de3b161-metrics-server-audit-profiles\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.710276 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.710250 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2f07ce17-c736-4e6f-bd59-f6c84de3b161-client-ca-bundle\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.710359 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.710283 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2f07ce17-c736-4e6f-bd59-f6c84de3b161-secret-metrics-server-tls\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.710467 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.710446 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2f07ce17-c736-4e6f-bd59-f6c84de3b161-secret-metrics-server-client-certs\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.723086 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.723060 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dxkl\" (UniqueName: \"kubernetes.io/projected/2f07ce17-c736-4e6f-bd59-f6c84de3b161-kube-api-access-5dxkl\") pod \"metrics-server-6b8df8fc77-n9zx4\" (UID: \"2f07ce17-c736-4e6f-bd59-f6c84de3b161\") " pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.803160 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.803120 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:08:46.972363 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.972338 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b8df8fc77-n9zx4"] Jul 22 16:08:46.974969 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:08:46.974941 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f07ce17_c736_4e6f_bd59_f6c84de3b161.slice/crio-680ae382a3774a06234d3cba674503113324840d81be002eba03c78328acbce3 WatchSource:0}: Error finding container 680ae382a3774a06234d3cba674503113324840d81be002eba03c78328acbce3: Status 404 returned error can't find the container with id 680ae382a3774a06234d3cba674503113324840d81be002eba03c78328acbce3 Jul 22 16:08:46.978672 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:46.978653 2567 scope.go:117] "RemoveContainer" containerID="f399f2ac5b0452a90ef782f2a358d615c92c9b73fe9dfd631783e97be5eaedc5" Jul 22 16:08:47.441438 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:47.441401 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" event={"ID":"2f07ce17-c736-4e6f-bd59-f6c84de3b161","Type":"ContainerStarted","Data":"680ae382a3774a06234d3cba674503113324840d81be002eba03c78328acbce3"} Jul 22 16:08:47.443219 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:47.443196 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:08:47.443331 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:47.443242 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" event={"ID":"799e2b87-1954-4103-860d-33ab42170349","Type":"ContainerStarted","Data":"f49c560f7e8afa3ef8ec4808305b91cf5427c3f52522679e2d439bd0289236d2"} Jul 22 16:08:47.443564 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:47.443544 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:08:47.465252 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:47.465188 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" podStartSLOduration=59.643118038 podStartE2EDuration="1m7.465165567s" totalImagesPullingTime="7.822047529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:47.463521144 +0000 UTC m=+92.605395247" watchObservedRunningTime="2026-07-22 16:08:47.465165567 +0000 UTC m=+92.607039645" Jul 22 16:08:47.515671 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:47.515640 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-sf2mn" Jul 22 16:08:49.452047 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:49.452009 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" event={"ID":"2f07ce17-c736-4e6f-bd59-f6c84de3b161","Type":"ContainerStarted","Data":"16fe969780d88a435b736dd33aaf4373edbf21da16a34effbde4dfa2337bebf5"} Jul 22 16:08:49.472798 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:49.472745 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" podStartSLOduration=1.7640814169999999 podStartE2EDuration="3.472726574s" totalImagesPullingTime="1.708645157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:08:49.470645128 +0000 UTC m=+94.612519218" watchObservedRunningTime="2026-07-22 16:08:49.472726574 +0000 UTC m=+94.614600650" Jul 22 16:08:52.538171 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:52.538120 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b465bb546-ppslg"] Jul 22 16:08:53.296006 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:08:53.295968 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rnnbk" Jul 22 16:09:06.803235 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:06.803194 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:09:06.803604 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:06.803260 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:09:10.531253 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:10.531219 2567 generic.go:356] "Generic (PLEG): container finished" podID="90abd0b8-4b88-4da8-838f-3202f3d8ad13" containerID="127d14a19301ae22c1ca51d5b7f8fb0abf0bb9df4e352924734899dd7939053c" exitCode=0 Jul 22 16:09:10.531604 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:10.531260 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" event={"ID":"90abd0b8-4b88-4da8-838f-3202f3d8ad13","Type":"ContainerDied","Data":"127d14a19301ae22c1ca51d5b7f8fb0abf0bb9df4e352924734899dd7939053c"} Jul 22 16:09:10.531604 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:10.531558 2567 scope.go:117] "RemoveContainer" containerID="127d14a19301ae22c1ca51d5b7f8fb0abf0bb9df4e352924734899dd7939053c" Jul 22 16:09:11.541494 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:11.541451 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zv9r5" event={"ID":"90abd0b8-4b88-4da8-838f-3202f3d8ad13","Type":"ContainerStarted","Data":"9a9c6a78e313869cfc67fcad56e85e593800730a6551cf767d05d9178a70f788"} Jul 22 16:09:17.558087 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.558025 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b465bb546-ppslg" podUID="1fe0626b-4d62-4845-b99c-9ff8d9e711e5" containerName="registry" containerID="cri-o://8b085b7c8b045bb347efe43f09c05c99a1e942a29c50e57e1c84c1c26408d0c1" gracePeriod=30 Jul 22 16:09:17.804331 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.804307 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:09:17.873029 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.872942 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-ca-trust-extracted\") pod \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " Jul 22 16:09:17.873029 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.872979 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-bound-sa-token\") pod \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " Jul 22 16:09:17.873253 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.873034 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-image-registry-private-configuration\") pod \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " Jul 22 16:09:17.873253 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.873060 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-trusted-ca\") pod \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " Jul 22 16:09:17.873253 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.873085 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls\") pod \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " Jul 22 16:09:17.873253 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.873102 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-certificates\") pod \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " Jul 22 16:09:17.873253 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.873153 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-installation-pull-secrets\") pod \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " Jul 22 16:09:17.873253 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.873175 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv95p\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-kube-api-access-dv95p\") pod \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\" (UID: \"1fe0626b-4d62-4845-b99c-9ff8d9e711e5\") " Jul 22 16:09:17.873590 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.873551 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1fe0626b-4d62-4845-b99c-9ff8d9e711e5" (UID: "1fe0626b-4d62-4845-b99c-9ff8d9e711e5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:09:17.873648 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.873616 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1fe0626b-4d62-4845-b99c-9ff8d9e711e5" (UID: "1fe0626b-4d62-4845-b99c-9ff8d9e711e5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:09:17.875629 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.875578 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1fe0626b-4d62-4845-b99c-9ff8d9e711e5" (UID: "1fe0626b-4d62-4845-b99c-9ff8d9e711e5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:09:17.875800 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.875667 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1fe0626b-4d62-4845-b99c-9ff8d9e711e5" (UID: "1fe0626b-4d62-4845-b99c-9ff8d9e711e5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:09:17.875800 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.875675 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1fe0626b-4d62-4845-b99c-9ff8d9e711e5" (UID: "1fe0626b-4d62-4845-b99c-9ff8d9e711e5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:09:17.875800 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.875729 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-kube-api-access-dv95p" (OuterVolumeSpecName: "kube-api-access-dv95p") pod "1fe0626b-4d62-4845-b99c-9ff8d9e711e5" (UID: "1fe0626b-4d62-4845-b99c-9ff8d9e711e5"). InnerVolumeSpecName "kube-api-access-dv95p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:09:17.875800 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.875775 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1fe0626b-4d62-4845-b99c-9ff8d9e711e5" (UID: "1fe0626b-4d62-4845-b99c-9ff8d9e711e5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:09:17.881422 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.881398 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1fe0626b-4d62-4845-b99c-9ff8d9e711e5" (UID: "1fe0626b-4d62-4845-b99c-9ff8d9e711e5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:09:17.974137 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.974094 2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-image-registry-private-configuration\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:09:17.974137 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.974128 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-trusted-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:09:17.974137 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.974137 2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:09:17.974137 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.974146 2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-registry-certificates\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:09:17.974386 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.974154 2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-installation-pull-secrets\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:09:17.974386 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.974163 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv95p\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-kube-api-access-dv95p\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:09:17.974386 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.974170 2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-ca-trust-extracted\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:09:17.974386 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:17.974178 2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fe0626b-4d62-4845-b99c-9ff8d9e711e5-bound-sa-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:09:18.569401 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:18.569364 2567 generic.go:356] "Generic (PLEG): container finished" podID="1fe0626b-4d62-4845-b99c-9ff8d9e711e5" containerID="8b085b7c8b045bb347efe43f09c05c99a1e942a29c50e57e1c84c1c26408d0c1" exitCode=0 Jul 22 16:09:18.569885 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:18.569413 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b465bb546-ppslg" Jul 22 16:09:18.569885 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:18.569453 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b465bb546-ppslg" event={"ID":"1fe0626b-4d62-4845-b99c-9ff8d9e711e5","Type":"ContainerDied","Data":"8b085b7c8b045bb347efe43f09c05c99a1e942a29c50e57e1c84c1c26408d0c1"} Jul 22 16:09:18.569885 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:18.569504 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b465bb546-ppslg" event={"ID":"1fe0626b-4d62-4845-b99c-9ff8d9e711e5","Type":"ContainerDied","Data":"40bb0dddcfb473c83f2ca5691c2cadc1592fd0e028afa9df4f2b1fb125a2f5cd"} Jul 22 16:09:18.569885 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:18.569524 2567 scope.go:117] "RemoveContainer" containerID="8b085b7c8b045bb347efe43f09c05c99a1e942a29c50e57e1c84c1c26408d0c1" Jul 22 16:09:18.577233 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:18.577221 2567 scope.go:117] "RemoveContainer" containerID="8b085b7c8b045bb347efe43f09c05c99a1e942a29c50e57e1c84c1c26408d0c1" Jul 22 16:09:18.577482 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:09:18.577463 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b085b7c8b045bb347efe43f09c05c99a1e942a29c50e57e1c84c1c26408d0c1\": container with ID starting with 8b085b7c8b045bb347efe43f09c05c99a1e942a29c50e57e1c84c1c26408d0c1 not found: ID does not exist" containerID="8b085b7c8b045bb347efe43f09c05c99a1e942a29c50e57e1c84c1c26408d0c1" Jul 22 16:09:18.577556 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:18.577495 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b085b7c8b045bb347efe43f09c05c99a1e942a29c50e57e1c84c1c26408d0c1"} err="failed to get container status \"8b085b7c8b045bb347efe43f09c05c99a1e942a29c50e57e1c84c1c26408d0c1\": rpc error: code = NotFound desc = could not find container \"8b085b7c8b045bb347efe43f09c05c99a1e942a29c50e57e1c84c1c26408d0c1\": container with ID starting with 8b085b7c8b045bb347efe43f09c05c99a1e942a29c50e57e1c84c1c26408d0c1 not found: ID does not exist" Jul 22 16:09:18.586057 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:18.586034 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b465bb546-ppslg"] Jul 22 16:09:18.594201 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:18.594177 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b465bb546-ppslg"] Jul 22 16:09:19.982255 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:19.982218 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fe0626b-4d62-4845-b99c-9ff8d9e711e5" path="/var/lib/kubelet/pods/1fe0626b-4d62-4845-b99c-9ff8d9e711e5/volumes" Jul 22 16:09:25.597827 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:25.597797 2567 generic.go:356] "Generic (PLEG): container finished" podID="957c1e60-c08f-437f-afaf-d3679b2e8376" containerID="055f78c9bd24e17d138701ff4fb1af7aa782ae1a7a0f8a9a545d3f3598f924b2" exitCode=0 Jul 22 16:09:25.598261 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:25.597870 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" event={"ID":"957c1e60-c08f-437f-afaf-d3679b2e8376","Type":"ContainerDied","Data":"055f78c9bd24e17d138701ff4fb1af7aa782ae1a7a0f8a9a545d3f3598f924b2"} Jul 22 16:09:25.598261 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:25.598214 2567 scope.go:117] "RemoveContainer" containerID="055f78c9bd24e17d138701ff4fb1af7aa782ae1a7a0f8a9a545d3f3598f924b2" Jul 22 16:09:25.599326 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:25.599307 2567 generic.go:356] "Generic (PLEG): container finished" podID="0f6ccc41-e229-40a9-9a3a-a6fa628dfe86" containerID="0a784101e3ecc4ca5c3331499a04e1b3995e09985dae7ead6ebb28e4a9746ae6" exitCode=0 Jul 22 16:09:25.599418 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:25.599361 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-24cbv" event={"ID":"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86","Type":"ContainerDied","Data":"0a784101e3ecc4ca5c3331499a04e1b3995e09985dae7ead6ebb28e4a9746ae6"} Jul 22 16:09:25.599623 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:25.599611 2567 scope.go:117] "RemoveContainer" containerID="0a784101e3ecc4ca5c3331499a04e1b3995e09985dae7ead6ebb28e4a9746ae6" Jul 22 16:09:26.605347 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:26.605309 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6s889" event={"ID":"957c1e60-c08f-437f-afaf-d3679b2e8376","Type":"ContainerStarted","Data":"7844a186d140d8571cfa39f5099d58a761075c5f7911b878f5c4218550a80ec6"} Jul 22 16:09:26.607150 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:26.607120 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-24cbv" event={"ID":"0f6ccc41-e229-40a9-9a3a-a6fa628dfe86","Type":"ContainerStarted","Data":"cdae8494e99e8fa5f89f739639dd68f10ce8cace18c03d6269ac8eda219fe9f0"} Jul 22 16:09:26.808886 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:26.808854 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:09:26.812783 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:09:26.812757 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6b8df8fc77-n9zx4" Jul 22 16:10:06.460616 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.460575 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-ff8cf4cc4-789d2"] Jul 22 16:10:06.461215 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.461046 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe0626b-4d62-4845-b99c-9ff8d9e711e5" containerName="registry" Jul 22 16:10:06.461215 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.461066 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe0626b-4d62-4845-b99c-9ff8d9e711e5" containerName="registry" Jul 22 16:10:06.461215 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.461153 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe0626b-4d62-4845-b99c-9ff8d9e711e5" containerName="registry" Jul 22 16:10:06.464250 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.464228 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.467152 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.467128 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 22 16:10:06.467853 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.467827 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 22 16:10:06.468094 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.467834 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kq6c4\"" Jul 22 16:10:06.468188 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.467876 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 22 16:10:06.468188 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.467920 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 22 16:10:06.468299 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.468208 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 22 16:10:06.476552 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.476526 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 22 16:10:06.480886 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.480854 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-ff8cf4cc4-789d2"] Jul 22 16:10:06.546958 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.546917 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3754bab-49a0-42b3-94f9-d875d57572ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.546958 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.546961 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3754bab-49a0-42b3-94f9-d875d57572ab-serving-certs-ca-bundle\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.547158 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.547007 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f3754bab-49a0-42b3-94f9-d875d57572ab-telemeter-client-tls\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.547158 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.547039 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3754bab-49a0-42b3-94f9-d875d57572ab-metrics-client-ca\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.547158 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.547106 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3754bab-49a0-42b3-94f9-d875d57572ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.547158 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.547130 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hptwk\" (UniqueName: \"kubernetes.io/projected/f3754bab-49a0-42b3-94f9-d875d57572ab-kube-api-access-hptwk\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.547274 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.547172 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f3754bab-49a0-42b3-94f9-d875d57572ab-federate-client-tls\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.547274 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.547189 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f3754bab-49a0-42b3-94f9-d875d57572ab-secret-telemeter-client\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.647677 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.647635 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f3754bab-49a0-42b3-94f9-d875d57572ab-federate-client-tls\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.647677 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.647675 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f3754bab-49a0-42b3-94f9-d875d57572ab-secret-telemeter-client\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.647954 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.647812 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3754bab-49a0-42b3-94f9-d875d57572ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.647954 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.647863 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3754bab-49a0-42b3-94f9-d875d57572ab-serving-certs-ca-bundle\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.647954 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.647893 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f3754bab-49a0-42b3-94f9-d875d57572ab-telemeter-client-tls\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.647954 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.647918 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3754bab-49a0-42b3-94f9-d875d57572ab-metrics-client-ca\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.648165 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.647980 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3754bab-49a0-42b3-94f9-d875d57572ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.648165 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.648007 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hptwk\" (UniqueName: \"kubernetes.io/projected/f3754bab-49a0-42b3-94f9-d875d57572ab-kube-api-access-hptwk\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.648745 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.648667 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3754bab-49a0-42b3-94f9-d875d57572ab-serving-certs-ca-bundle\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.648934 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.648912 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3754bab-49a0-42b3-94f9-d875d57572ab-metrics-client-ca\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.649064 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.649041 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3754bab-49a0-42b3-94f9-d875d57572ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.650873 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.650825 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3754bab-49a0-42b3-94f9-d875d57572ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.650986 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.650912 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f3754bab-49a0-42b3-94f9-d875d57572ab-federate-client-tls\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.651046 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.650998 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f3754bab-49a0-42b3-94f9-d875d57572ab-secret-telemeter-client\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.651046 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.651011 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f3754bab-49a0-42b3-94f9-d875d57572ab-telemeter-client-tls\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.659328 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.659288 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hptwk\" (UniqueName: \"kubernetes.io/projected/f3754bab-49a0-42b3-94f9-d875d57572ab-kube-api-access-hptwk\") pod \"telemeter-client-ff8cf4cc4-789d2\" (UID: \"f3754bab-49a0-42b3-94f9-d875d57572ab\") " pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.776565 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.776433 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" Jul 22 16:10:06.938760 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:06.938679 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-ff8cf4cc4-789d2"] Jul 22 16:10:06.943521 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:10:06.943489 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3754bab_49a0_42b3_94f9_d875d57572ab.slice/crio-7d379a30baf27f370a03157f45c7ebfc53ec3fb581c3c62d544bcd40666c3ef6 WatchSource:0}: Error finding container 7d379a30baf27f370a03157f45c7ebfc53ec3fb581c3c62d544bcd40666c3ef6: Status 404 returned error can't find the container with id 7d379a30baf27f370a03157f45c7ebfc53ec3fb581c3c62d544bcd40666c3ef6 Jul 22 16:10:07.765356 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:07.765318 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" event={"ID":"f3754bab-49a0-42b3-94f9-d875d57572ab","Type":"ContainerStarted","Data":"7d379a30baf27f370a03157f45c7ebfc53ec3fb581c3c62d544bcd40666c3ef6"} Jul 22 16:10:09.774635 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:09.774599 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" event={"ID":"f3754bab-49a0-42b3-94f9-d875d57572ab","Type":"ContainerStarted","Data":"e79f85b48360d5819b1a3542897685f6c213f279a1917e7c4bf04de29d713a34"} Jul 22 16:10:10.780582 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:10.780545 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" event={"ID":"f3754bab-49a0-42b3-94f9-d875d57572ab","Type":"ContainerStarted","Data":"81cdab867aab2e07df84877bb57c20d04bcbbf1c1b727f2bd7cc24290a4e0c53"} Jul 22 16:10:10.780972 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:10.780609 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" event={"ID":"f3754bab-49a0-42b3-94f9-d875d57572ab","Type":"ContainerStarted","Data":"d31f712350054c2fc5a777809a26724433b1427330c446414c4e218c2cad1f80"} Jul 22 16:10:10.806410 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:10:10.806349 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-ff8cf4cc4-789d2" podStartSLOduration=1.530644037 podStartE2EDuration="4.806328806s" totalImagesPullingTime="3.275684769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:10:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:10:10.804189114 +0000 UTC m=+175.946063201" watchObservedRunningTime="2026-07-22 16:10:10.806328806 +0000 UTC m=+175.948202883" Jul 22 16:12:15.820480 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:15.820452 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:12:15.824436 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:15.824413 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:12:15.826822 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:15.826802 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:12:15.830988 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:15.830972 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:12:15.831374 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:15.831355 2567 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 16:12:55.399515 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.399485 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xh2gw"] Jul 22 16:12:55.402445 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.402424 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xh2gw" Jul 22 16:12:55.407054 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.407034 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 22 16:12:55.407151 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.407133 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-l4fbg\"" Jul 22 16:12:55.408751 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.408734 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:12:55.524949 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.524894 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc5d868c-9d42-4540-a4b5-963d8d9b16cb-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xh2gw\" (UID: \"fc5d868c-9d42-4540-a4b5-963d8d9b16cb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xh2gw" Jul 22 16:12:55.524949 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.524952 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnr8r\" (UniqueName: \"kubernetes.io/projected/fc5d868c-9d42-4540-a4b5-963d8d9b16cb-kube-api-access-wnr8r\") pod \"cert-manager-operator-controller-manager-55c444bf86-xh2gw\" (UID: \"fc5d868c-9d42-4540-a4b5-963d8d9b16cb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xh2gw" Jul 22 16:12:55.527295 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.527265 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xh2gw"] Jul 22 16:12:55.625595 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.625562 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc5d868c-9d42-4540-a4b5-963d8d9b16cb-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xh2gw\" (UID: \"fc5d868c-9d42-4540-a4b5-963d8d9b16cb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xh2gw" Jul 22 16:12:55.625595 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.625602 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnr8r\" (UniqueName: \"kubernetes.io/projected/fc5d868c-9d42-4540-a4b5-963d8d9b16cb-kube-api-access-wnr8r\") pod \"cert-manager-operator-controller-manager-55c444bf86-xh2gw\" (UID: \"fc5d868c-9d42-4540-a4b5-963d8d9b16cb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xh2gw" Jul 22 16:12:55.626054 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.626032 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc5d868c-9d42-4540-a4b5-963d8d9b16cb-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xh2gw\" (UID: \"fc5d868c-9d42-4540-a4b5-963d8d9b16cb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xh2gw" Jul 22 16:12:55.641232 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.641207 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnr8r\" (UniqueName: \"kubernetes.io/projected/fc5d868c-9d42-4540-a4b5-963d8d9b16cb-kube-api-access-wnr8r\") pod \"cert-manager-operator-controller-manager-55c444bf86-xh2gw\" (UID: \"fc5d868c-9d42-4540-a4b5-963d8d9b16cb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xh2gw" Jul 22 16:12:55.711244 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.711160 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xh2gw" Jul 22 16:12:55.872418 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.872395 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xh2gw"] Jul 22 16:12:55.875062 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:12:55.875035 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5d868c_9d42_4540_a4b5_963d8d9b16cb.slice/crio-cdd8005b910f8d8b26eb7f3e012959defdf602931f58cea79d03e2d9b8ec009c WatchSource:0}: Error finding container cdd8005b910f8d8b26eb7f3e012959defdf602931f58cea79d03e2d9b8ec009c: Status 404 returned error can't find the container with id cdd8005b910f8d8b26eb7f3e012959defdf602931f58cea79d03e2d9b8ec009c Jul 22 16:12:55.877616 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:55.877597 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:12:56.398345 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:56.398311 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xh2gw" event={"ID":"fc5d868c-9d42-4540-a4b5-963d8d9b16cb","Type":"ContainerStarted","Data":"cdd8005b910f8d8b26eb7f3e012959defdf602931f58cea79d03e2d9b8ec009c"} Jul 22 16:12:59.413347 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:59.413313 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xh2gw" event={"ID":"fc5d868c-9d42-4540-a4b5-963d8d9b16cb","Type":"ContainerStarted","Data":"e6fd357ef849d669106ecf959f046a5ede6fb31e45337c9ef4bb48704002fd0d"} Jul 22 16:12:59.457950 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:59.457894 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xh2gw" podStartSLOduration=1.7153459020000001 podStartE2EDuration="4.457879038s" totalImagesPullingTime="2.742533136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:12:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:12:59.457245489 +0000 UTC m=+344.599119564" watchObservedRunningTime="2026-07-22 16:12:59.457879038 +0000 UTC m=+344.599753114" Jul 22 16:12:59.964346 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:59.964311 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-njpjg"] Jul 22 16:12:59.967581 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:59.967560 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-njpjg" Jul 22 16:12:59.976245 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:12:59.976220 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 16:13:00.011133 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:00.011096 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-njpjg"] Jul 22 16:13:00.013873 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:00.013851 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 16:13:00.067274 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:00.067236 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d519eac4-271f-4f29-aa7d-eb6b50c10581-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-njpjg\" (UID: \"d519eac4-271f-4f29-aa7d-eb6b50c10581\") " pod="cert-manager/cert-manager-webhook-6486ffd997-njpjg" Jul 22 16:13:00.067445 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:00.067304 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwpkf\" (UniqueName: \"kubernetes.io/projected/d519eac4-271f-4f29-aa7d-eb6b50c10581-kube-api-access-qwpkf\") pod \"cert-manager-webhook-6486ffd997-njpjg\" (UID: \"d519eac4-271f-4f29-aa7d-eb6b50c10581\") " pod="cert-manager/cert-manager-webhook-6486ffd997-njpjg" Jul 22 16:13:00.168634 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:00.168597 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwpkf\" (UniqueName: \"kubernetes.io/projected/d519eac4-271f-4f29-aa7d-eb6b50c10581-kube-api-access-qwpkf\") pod \"cert-manager-webhook-6486ffd997-njpjg\" (UID: \"d519eac4-271f-4f29-aa7d-eb6b50c10581\") " pod="cert-manager/cert-manager-webhook-6486ffd997-njpjg" Jul 22 16:13:00.168813 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:00.168645 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d519eac4-271f-4f29-aa7d-eb6b50c10581-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-njpjg\" (UID: \"d519eac4-271f-4f29-aa7d-eb6b50c10581\") " pod="cert-manager/cert-manager-webhook-6486ffd997-njpjg" Jul 22 16:13:00.201232 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:00.201196 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwpkf\" (UniqueName: \"kubernetes.io/projected/d519eac4-271f-4f29-aa7d-eb6b50c10581-kube-api-access-qwpkf\") pod \"cert-manager-webhook-6486ffd997-njpjg\" (UID: \"d519eac4-271f-4f29-aa7d-eb6b50c10581\") " pod="cert-manager/cert-manager-webhook-6486ffd997-njpjg" Jul 22 16:13:00.206020 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:00.205992 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d519eac4-271f-4f29-aa7d-eb6b50c10581-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-njpjg\" (UID: \"d519eac4-271f-4f29-aa7d-eb6b50c10581\") " pod="cert-manager/cert-manager-webhook-6486ffd997-njpjg" Jul 22 16:13:00.289920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:00.289885 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-njpjg" Jul 22 16:13:00.477846 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:00.477814 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-njpjg"] Jul 22 16:13:00.478859 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:13:00.478836 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd519eac4_271f_4f29_aa7d_eb6b50c10581.slice/crio-8cff2a9e305823ebef527e5abaaa14b0bcef60b9ba11753923167d3467d50216 WatchSource:0}: Error finding container 8cff2a9e305823ebef527e5abaaa14b0bcef60b9ba11753923167d3467d50216: Status 404 returned error can't find the container with id 8cff2a9e305823ebef527e5abaaa14b0bcef60b9ba11753923167d3467d50216 Jul 22 16:13:01.423843 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:01.423801 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-njpjg" event={"ID":"d519eac4-271f-4f29-aa7d-eb6b50c10581","Type":"ContainerStarted","Data":"8cff2a9e305823ebef527e5abaaa14b0bcef60b9ba11753923167d3467d50216"} Jul 22 16:13:03.439716 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:03.439606 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-njpjg" event={"ID":"d519eac4-271f-4f29-aa7d-eb6b50c10581","Type":"ContainerStarted","Data":"f948d9ffc9bcaa82624c2b1a7aeafa4e6b2a273af3f066ffd7101b3a0ac7fb5d"} Jul 22 16:13:03.439716 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:03.439669 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-njpjg" Jul 22 16:13:09.446203 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:09.446127 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-njpjg" Jul 22 16:13:09.502105 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:09.502052 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-njpjg" podStartSLOduration=7.932350318 podStartE2EDuration="10.502038158s" totalImagesPullingTime="2.56968784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:12:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:03.503753781 +0000 UTC m=+348.645627858" watchObservedRunningTime="2026-07-22 16:13:09.502038158 +0000 UTC m=+354.643912235" Jul 22 16:13:14.080219 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:14.080182 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dcnv"] Jul 22 16:13:14.083450 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:14.083429 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dcnv" Jul 22 16:13:14.096114 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:14.096089 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 16:13:14.096265 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:14.096146 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:13:14.096739 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:14.096723 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-wc6d5\"" Jul 22 16:13:14.106295 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:14.106263 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dcnv"] Jul 22 16:13:14.194671 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:14.194636 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9dw\" (UniqueName: \"kubernetes.io/projected/ff65214b-edb6-4ea4-abf1-c690520ae97f-kube-api-access-hf9dw\") pod \"openshift-lws-operator-fd8ccff4c-9dcnv\" (UID: \"ff65214b-edb6-4ea4-abf1-c690520ae97f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dcnv" Jul 22 16:13:14.194671 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:14.194672 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff65214b-edb6-4ea4-abf1-c690520ae97f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9dcnv\" (UID: \"ff65214b-edb6-4ea4-abf1-c690520ae97f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dcnv" Jul 22 16:13:14.296134 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:14.296102 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9dw\" (UniqueName: \"kubernetes.io/projected/ff65214b-edb6-4ea4-abf1-c690520ae97f-kube-api-access-hf9dw\") pod \"openshift-lws-operator-fd8ccff4c-9dcnv\" (UID: \"ff65214b-edb6-4ea4-abf1-c690520ae97f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dcnv" Jul 22 16:13:14.296134 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:14.296137 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff65214b-edb6-4ea4-abf1-c690520ae97f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9dcnv\" (UID: \"ff65214b-edb6-4ea4-abf1-c690520ae97f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dcnv" Jul 22 16:13:14.296586 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:14.296566 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff65214b-edb6-4ea4-abf1-c690520ae97f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9dcnv\" (UID: \"ff65214b-edb6-4ea4-abf1-c690520ae97f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dcnv" Jul 22 16:13:14.309661 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:14.309632 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9dw\" (UniqueName: \"kubernetes.io/projected/ff65214b-edb6-4ea4-abf1-c690520ae97f-kube-api-access-hf9dw\") pod \"openshift-lws-operator-fd8ccff4c-9dcnv\" (UID: \"ff65214b-edb6-4ea4-abf1-c690520ae97f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dcnv" Jul 22 16:13:14.392444 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:14.392361 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dcnv" Jul 22 16:13:14.551952 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:14.551918 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dcnv"] Jul 22 16:13:14.553772 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:13:14.553712 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff65214b_edb6_4ea4_abf1_c690520ae97f.slice/crio-b048becf46bf62a699f1197f0142ca7379508a480f8ddcd576d7bc17d57a99fe WatchSource:0}: Error finding container b048becf46bf62a699f1197f0142ca7379508a480f8ddcd576d7bc17d57a99fe: Status 404 returned error can't find the container with id b048becf46bf62a699f1197f0142ca7379508a480f8ddcd576d7bc17d57a99fe Jul 22 16:13:15.489985 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:15.489950 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dcnv" event={"ID":"ff65214b-edb6-4ea4-abf1-c690520ae97f","Type":"ContainerStarted","Data":"b048becf46bf62a699f1197f0142ca7379508a480f8ddcd576d7bc17d57a99fe"} Jul 22 16:13:18.505352 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:18.505316 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dcnv" event={"ID":"ff65214b-edb6-4ea4-abf1-c690520ae97f","Type":"ContainerStarted","Data":"fa656c6a9c957f5cbbae767b7238e6e487e7e50392bcaa82f56bcad55447a54d"} Jul 22 16:13:34.876781 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:34.876726 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dcnv" podStartSLOduration=17.701540184 podStartE2EDuration="20.876707868s" totalImagesPullingTime="3.175167684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:18.56617893 +0000 UTC m=+363.708053007" watchObservedRunningTime="2026-07-22 16:13:34.876707868 +0000 UTC m=+380.018581939" Jul 22 16:13:34.877291 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:34.877199 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw"] Jul 22 16:13:34.949661 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:34.949621 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw"] Jul 22 16:13:34.949849 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:34.949738 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:34.960609 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:34.960584 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x4fjv\"" Jul 22 16:13:34.966180 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:34.966155 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/239e7a15-399f-4a8a-8341-edb28cf3faa4-tmp\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:34.966304 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:34.966198 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/239e7a15-399f-4a8a-8341-edb28cf3faa4-manifests\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:34.966304 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:34.966226 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24z9z\" (UniqueName: \"kubernetes.io/projected/239e7a15-399f-4a8a-8341-edb28cf3faa4-kube-api-access-24z9z\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:34.966304 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:34.966273 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/239e7a15-399f-4a8a-8341-edb28cf3faa4-webhook-cert\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:34.966441 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:34.966324 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/239e7a15-399f-4a8a-8341-edb28cf3faa4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:35.003266 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.003232 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 16:13:35.024103 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.024075 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 16:13:35.031485 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.031417 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 16:13:35.031664 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.031499 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 16:13:35.067522 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.067482 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/239e7a15-399f-4a8a-8341-edb28cf3faa4-tmp\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:35.067522 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.067523 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/239e7a15-399f-4a8a-8341-edb28cf3faa4-manifests\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:35.067750 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.067545 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24z9z\" (UniqueName: \"kubernetes.io/projected/239e7a15-399f-4a8a-8341-edb28cf3faa4-kube-api-access-24z9z\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:35.067750 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.067578 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/239e7a15-399f-4a8a-8341-edb28cf3faa4-webhook-cert\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:35.067750 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.067600 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/239e7a15-399f-4a8a-8341-edb28cf3faa4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:35.068022 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.068002 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/239e7a15-399f-4a8a-8341-edb28cf3faa4-manifests\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:35.070343 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.070317 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/239e7a15-399f-4a8a-8341-edb28cf3faa4-tmp\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:35.070503 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.070479 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/239e7a15-399f-4a8a-8341-edb28cf3faa4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:35.070569 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.070544 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/239e7a15-399f-4a8a-8341-edb28cf3faa4-webhook-cert\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:35.099594 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.099561 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24z9z\" (UniqueName: \"kubernetes.io/projected/239e7a15-399f-4a8a-8341-edb28cf3faa4-kube-api-access-24z9z\") pod \"opendatahub-operator-controller-manager-587bf8b54-qg9rw\" (UID: \"239e7a15-399f-4a8a-8341-edb28cf3faa4\") " pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:35.262313 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.262230 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:35.441529 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.441489 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw"] Jul 22 16:13:35.444967 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:13:35.444931 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239e7a15_399f_4a8a_8341_edb28cf3faa4.slice/crio-1801d2013ac5d8a54123189c9043d6803a8dc924b45a42ddfb5ac9b00f8e62af WatchSource:0}: Error finding container 1801d2013ac5d8a54123189c9043d6803a8dc924b45a42ddfb5ac9b00f8e62af: Status 404 returned error can't find the container with id 1801d2013ac5d8a54123189c9043d6803a8dc924b45a42ddfb5ac9b00f8e62af Jul 22 16:13:35.576965 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:35.576911 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" event={"ID":"239e7a15-399f-4a8a-8341-edb28cf3faa4","Type":"ContainerStarted","Data":"1801d2013ac5d8a54123189c9043d6803a8dc924b45a42ddfb5ac9b00f8e62af"} Jul 22 16:13:36.400315 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.400272 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs"] Jul 22 16:13:36.402599 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.402575 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:36.407482 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.407354 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2tkjq\"" Jul 22 16:13:36.407482 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.407354 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 16:13:36.407675 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.407362 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 16:13:36.408000 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.407970 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 16:13:36.452617 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.452539 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs"] Jul 22 16:13:36.479127 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.479085 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md56b\" (UniqueName: \"kubernetes.io/projected/858a6d87-46bd-43b9-8056-3b05c72113c6-kube-api-access-md56b\") pod \"lws-controller-manager-b9d69d95c-tlvgs\" (UID: \"858a6d87-46bd-43b9-8056-3b05c72113c6\") " pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:36.479299 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.479139 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/858a6d87-46bd-43b9-8056-3b05c72113c6-cert\") pod \"lws-controller-manager-b9d69d95c-tlvgs\" (UID: \"858a6d87-46bd-43b9-8056-3b05c72113c6\") " pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:36.479299 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.479200 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/858a6d87-46bd-43b9-8056-3b05c72113c6-metrics-cert\") pod \"lws-controller-manager-b9d69d95c-tlvgs\" (UID: \"858a6d87-46bd-43b9-8056-3b05c72113c6\") " pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:36.479299 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.479249 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/858a6d87-46bd-43b9-8056-3b05c72113c6-manager-config\") pod \"lws-controller-manager-b9d69d95c-tlvgs\" (UID: \"858a6d87-46bd-43b9-8056-3b05c72113c6\") " pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:36.580335 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.580300 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/858a6d87-46bd-43b9-8056-3b05c72113c6-metrics-cert\") pod \"lws-controller-manager-b9d69d95c-tlvgs\" (UID: \"858a6d87-46bd-43b9-8056-3b05c72113c6\") " pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:36.580518 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.580360 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/858a6d87-46bd-43b9-8056-3b05c72113c6-manager-config\") pod \"lws-controller-manager-b9d69d95c-tlvgs\" (UID: \"858a6d87-46bd-43b9-8056-3b05c72113c6\") " pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:36.580518 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.580429 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md56b\" (UniqueName: \"kubernetes.io/projected/858a6d87-46bd-43b9-8056-3b05c72113c6-kube-api-access-md56b\") pod \"lws-controller-manager-b9d69d95c-tlvgs\" (UID: \"858a6d87-46bd-43b9-8056-3b05c72113c6\") " pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:36.580518 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.580458 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/858a6d87-46bd-43b9-8056-3b05c72113c6-cert\") pod \"lws-controller-manager-b9d69d95c-tlvgs\" (UID: \"858a6d87-46bd-43b9-8056-3b05c72113c6\") " pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:36.581267 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.581216 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/858a6d87-46bd-43b9-8056-3b05c72113c6-manager-config\") pod \"lws-controller-manager-b9d69d95c-tlvgs\" (UID: \"858a6d87-46bd-43b9-8056-3b05c72113c6\") " pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:36.583771 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.583746 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/858a6d87-46bd-43b9-8056-3b05c72113c6-cert\") pod \"lws-controller-manager-b9d69d95c-tlvgs\" (UID: \"858a6d87-46bd-43b9-8056-3b05c72113c6\") " pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:36.584381 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.584356 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/858a6d87-46bd-43b9-8056-3b05c72113c6-metrics-cert\") pod \"lws-controller-manager-b9d69d95c-tlvgs\" (UID: \"858a6d87-46bd-43b9-8056-3b05c72113c6\") " pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:36.609563 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.609505 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md56b\" (UniqueName: \"kubernetes.io/projected/858a6d87-46bd-43b9-8056-3b05c72113c6-kube-api-access-md56b\") pod \"lws-controller-manager-b9d69d95c-tlvgs\" (UID: \"858a6d87-46bd-43b9-8056-3b05c72113c6\") " pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:36.714518 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.714436 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:36.902448 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:36.902407 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs"] Jul 22 16:13:36.905858 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:13:36.905819 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858a6d87_46bd_43b9_8056_3b05c72113c6.slice/crio-8a0b4f172f2db1d9afb9e58c71bc2f4650e3da699337388cc224ce5a3ef40405 WatchSource:0}: Error finding container 8a0b4f172f2db1d9afb9e58c71bc2f4650e3da699337388cc224ce5a3ef40405: Status 404 returned error can't find the container with id 8a0b4f172f2db1d9afb9e58c71bc2f4650e3da699337388cc224ce5a3ef40405 Jul 22 16:13:37.589382 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:37.589326 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" event={"ID":"858a6d87-46bd-43b9-8056-3b05c72113c6","Type":"ContainerStarted","Data":"8a0b4f172f2db1d9afb9e58c71bc2f4650e3da699337388cc224ce5a3ef40405"} Jul 22 16:13:39.600999 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:39.600951 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" event={"ID":"858a6d87-46bd-43b9-8056-3b05c72113c6","Type":"ContainerStarted","Data":"3c4ef16584ea11d39a7d186d694c7934408192af33bebb5596f89e9518b4ebe9"} Jul 22 16:13:39.601402 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:39.601043 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:39.602667 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:39.602643 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" event={"ID":"239e7a15-399f-4a8a-8341-edb28cf3faa4","Type":"ContainerStarted","Data":"6673ba31c10d043ded5a9c0ab42e51e7fb2b4aa2a79e4c37fed896ee6b5e5244"} Jul 22 16:13:39.658913 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:39.658813 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" podStartSLOduration=1.184563186 podStartE2EDuration="3.658797973s" totalImagesPullingTime="2.474234787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:39.658434787 +0000 UTC m=+384.800308863" watchObservedRunningTime="2026-07-22 16:13:39.658797973 +0000 UTC m=+384.800672118" Jul 22 16:13:40.608727 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:40.608672 2567 generic.go:356] "Generic (PLEG): container finished" podID="239e7a15-399f-4a8a-8341-edb28cf3faa4" containerID="6673ba31c10d043ded5a9c0ab42e51e7fb2b4aa2a79e4c37fed896ee6b5e5244" exitCode=0 Jul 22 16:13:40.609143 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:40.608760 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" event={"ID":"239e7a15-399f-4a8a-8341-edb28cf3faa4","Type":"ContainerDied","Data":"6673ba31c10d043ded5a9c0ab42e51e7fb2b4aa2a79e4c37fed896ee6b5e5244"} Jul 22 16:13:41.615248 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:41.615212 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" event={"ID":"239e7a15-399f-4a8a-8341-edb28cf3faa4","Type":"ContainerStarted","Data":"e05e54e39ea50b08c6dc15ac35fb3f569581e5dfcbdeab99694eeb97edc014c0"} Jul 22 16:13:41.615635 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:41.615355 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:13:41.659041 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:41.653933 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" podStartSLOduration=2.86447921 podStartE2EDuration="7.653905377s" totalImagesPullingTime="3.49209295s" totalInitContainerRuntime="1.297333217s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:13:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:41.652040422 +0000 UTC m=+386.793914499" watchObservedRunningTime="2026-07-22 16:13:41.653905377 +0000 UTC m=+386.795779456" Jul 22 16:13:50.610941 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:50.610910 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-b9d69d95c-tlvgs" Jul 22 16:13:52.622285 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:13:52.622257 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-587bf8b54-qg9rw" Jul 22 16:14:08.136387 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.136351 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh"] Jul 22 16:14:08.139429 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.139411 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.152115 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:14:08.152088 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"data-science-gateway-data-science-gateway-class-dockercfg-zztgz\" is forbidden: User \"system:node:ip-10-0-131-21.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-131-21.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zztgz\"" type="*v1.Secret" Jul 22 16:14:08.152227 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:14:08.152128 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" is forbidden: User \"system:node:ip-10-0-131-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-131-21.ec2.internal' and this object" podUID="85f1c726-74bd-42a4-8f0d-342ac1e83aad" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.153825 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:14:08.153804 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-gw-ca-root-cert\" is forbidden: User \"system:node:ip-10-0-131-21.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-131-21.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" Jul 22 16:14:08.193261 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.193231 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh"] Jul 22 16:14:08.267150 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.267111 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.267319 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.267161 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.267319 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.267248 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.267319 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.267280 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.267319 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.267298 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.267460 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.267329 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.267460 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.267358 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.267460 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.267415 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.267460 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.267433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knpnf\" (UniqueName: \"kubernetes.io/projected/85f1c726-74bd-42a4-8f0d-342ac1e83aad-kube-api-access-knpnf\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.368569 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.368532 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.368765 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.368585 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.368765 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.368616 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.368765 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.368634 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.368765 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.368655 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.368765 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.368682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.368765 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.368729 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.369076 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.368785 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.369076 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.368812 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knpnf\" (UniqueName: \"kubernetes.io/projected/85f1c726-74bd-42a4-8f0d-342ac1e83aad-kube-api-access-knpnf\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.369177 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.369114 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.369228 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.369193 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.369276 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.369232 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.369320 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.369287 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.370995 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.370977 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.371259 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.371236 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.415394 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.415321 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knpnf\" (UniqueName: \"kubernetes.io/projected/85f1c726-74bd-42a4-8f0d-342ac1e83aad-kube-api-access-knpnf\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:08.452482 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:08.452443 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:09.369891 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:14:09.369858 2567 configmap.go:193] Couldn't get configMap openshift-ingress/openshift-gw-ca-root-cert: failed to sync configmap cache: timed out waiting for the condition Jul 22 16:14:09.370243 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:14:09.369939 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istiod-ca-cert podName:85f1c726-74bd-42a4-8f0d-342ac1e83aad nodeName:}" failed. No retries permitted until 2026-07-22 16:14:09.86992285 +0000 UTC m=+415.011796905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "istiod-ca-cert" (UniqueName: "kubernetes.io/configmap/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istiod-ca-cert") pod "data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" (UID: "85f1c726-74bd-42a4-8f0d-342ac1e83aad") : failed to sync configmap cache: timed out waiting for the condition Jul 22 16:14:09.444726 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.444671 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zztgz\"" Jul 22 16:14:09.600135 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.600102 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7"] Jul 22 16:14:09.604384 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.604366 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.611477 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.611457 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 16:14:09.611591 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.611462 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 16:14:09.611654 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.611624 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 16:14:09.611748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.611732 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 16:14:09.611823 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.611758 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zxf5w\"" Jul 22 16:14:09.638969 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.638897 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7"] Jul 22 16:14:09.779309 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.779269 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/252f357d-2451-4e46-9887-075ccae6dd38-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.779505 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.779326 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/252f357d-2451-4e46-9887-075ccae6dd38-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.779505 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.779408 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/252f357d-2451-4e46-9887-075ccae6dd38-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.779505 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.779456 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/252f357d-2451-4e46-9887-075ccae6dd38-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.779661 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.779507 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvhqg\" (UniqueName: \"kubernetes.io/projected/252f357d-2451-4e46-9887-075ccae6dd38-kube-api-access-dvhqg\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.779661 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.779533 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/252f357d-2451-4e46-9887-075ccae6dd38-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.779661 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.779583 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/252f357d-2451-4e46-9887-075ccae6dd38-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.880176 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.880136 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/252f357d-2451-4e46-9887-075ccae6dd38-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.880339 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.880194 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/252f357d-2451-4e46-9887-075ccae6dd38-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.880339 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.880229 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/252f357d-2451-4e46-9887-075ccae6dd38-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.880339 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.880264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvhqg\" (UniqueName: \"kubernetes.io/projected/252f357d-2451-4e46-9887-075ccae6dd38-kube-api-access-dvhqg\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.880452 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.880396 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/252f357d-2451-4e46-9887-075ccae6dd38-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.880452 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.880441 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:09.880554 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.880476 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/252f357d-2451-4e46-9887-075ccae6dd38-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.880554 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.880532 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/252f357d-2451-4e46-9887-075ccae6dd38-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.881148 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.881124 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/252f357d-2451-4e46-9887-075ccae6dd38-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.881280 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.881149 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f1c726-74bd-42a4-8f0d-342ac1e83aad-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678ndvrh\" (UID: \"85f1c726-74bd-42a4-8f0d-342ac1e83aad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:09.882657 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.882638 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/252f357d-2451-4e46-9887-075ccae6dd38-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.882888 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.882872 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/252f357d-2451-4e46-9887-075ccae6dd38-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.883053 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.883034 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/252f357d-2451-4e46-9887-075ccae6dd38-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.883109 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.883056 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/252f357d-2451-4e46-9887-075ccae6dd38-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.894060 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.894005 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/252f357d-2451-4e46-9887-075ccae6dd38-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.894376 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.894359 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvhqg\" (UniqueName: \"kubernetes.io/projected/252f357d-2451-4e46-9887-075ccae6dd38-kube-api-access-dvhqg\") pod \"istiod-openshift-gateway-94bb8fbfd-8d5b7\" (UID: \"252f357d-2451-4e46-9887-075ccae6dd38\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.914329 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.914309 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:09.948589 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:09.948550 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:10.191913 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:14:10.191861 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f1c726_74bd_42a4_8f0d_342ac1e83aad.slice/crio-147b63f26b8a111a7e0edec05795f374de9372c6e9e143e700708deeca912538 WatchSource:0}: Error finding container 147b63f26b8a111a7e0edec05795f374de9372c6e9e143e700708deeca912538: Status 404 returned error can't find the container with id 147b63f26b8a111a7e0edec05795f374de9372c6e9e143e700708deeca912538 Jul 22 16:14:10.193581 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:10.193510 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh"] Jul 22 16:14:10.221394 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:10.221368 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7"] Jul 22 16:14:10.248566 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:14:10.248540 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252f357d_2451_4e46_9887_075ccae6dd38.slice/crio-3cc65158b9a69e84272d85de261044ca8240ced4a2af79b61df49599688074ac WatchSource:0}: Error finding container 3cc65158b9a69e84272d85de261044ca8240ced4a2af79b61df49599688074ac: Status 404 returned error can't find the container with id 3cc65158b9a69e84272d85de261044ca8240ced4a2af79b61df49599688074ac Jul 22 16:14:10.742891 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:10.742857 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" event={"ID":"252f357d-2451-4e46-9887-075ccae6dd38","Type":"ContainerStarted","Data":"3cc65158b9a69e84272d85de261044ca8240ced4a2af79b61df49599688074ac"} Jul 22 16:14:10.743863 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:10.743840 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" event={"ID":"85f1c726-74bd-42a4-8f0d-342ac1e83aad","Type":"ContainerStarted","Data":"147b63f26b8a111a7e0edec05795f374de9372c6e9e143e700708deeca912538"} Jul 22 16:14:13.048906 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:13.048870 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:14:13.049159 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:13.048966 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:14:13.049159 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:13.048997 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:14:13.759779 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:13.759729 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" event={"ID":"85f1c726-74bd-42a4-8f0d-342ac1e83aad","Type":"ContainerStarted","Data":"df2b66efa3318d069bc560aa0cfe27cc79a2660a2f800534a1a2bf8ce0c193ed"} Jul 22 16:14:13.877747 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:13.877673 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" podStartSLOduration=3.022987323 podStartE2EDuration="5.877652787s" totalImagesPullingTime="2.854665464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:14:13.877412957 +0000 UTC m=+419.019287063" watchObservedRunningTime="2026-07-22 16:14:13.877652787 +0000 UTC m=+419.019526864" Jul 22 16:14:13.948830 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:13.948758 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:14.826735 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:14.826679 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:14:14.827010 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:14.826764 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:14:14.953394 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:14.953365 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:15.771072 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:15.771034 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" event={"ID":"252f357d-2451-4e46-9887-075ccae6dd38","Type":"ContainerStarted","Data":"7c22990a028485eecbaabf5cc65f42afb026976de486a99c0b46bc0c3654e3f9"} Jul 22 16:14:15.771270 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:15.771223 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:15.771270 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:15.771269 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:14:15.772076 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:15.772056 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678ndvrh" Jul 22 16:14:15.772704 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:15.772664 2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-8d5b7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 16:14:15.772794 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:15.772740 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" podUID="252f357d-2451-4e46-9887-075ccae6dd38" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 16:14:15.967279 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:15.967225 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" podStartSLOduration=2.391284836 podStartE2EDuration="6.967210503s" totalImagesPullingTime="4.575925667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:14:15.832244912 +0000 UTC m=+420.974118988" watchObservedRunningTime="2026-07-22 16:14:15.967210503 +0000 UTC m=+421.109084627" Jul 22 16:14:16.776640 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:16.776599 2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-8d5b7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 16:14:16.776819 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:16.776655 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" podUID="252f357d-2451-4e46-9887-075ccae6dd38" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 16:14:19.776953 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:14:19.776917 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8d5b7" Jul 22 16:16:43.342920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.342849 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:16:43.345129 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.345111 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nhksq" Jul 22 16:16:43.347900 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.347881 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vckdj\"" Jul 22 16:16:43.348248 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.348229 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 16:16:43.348733 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.348712 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 16:16:43.348837 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.348743 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 16:16:43.358186 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.358165 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:16:43.389699 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.389662 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:16:43.461097 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.461065 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/704997ce-7891-4120-a9ca-a114f41ed771-config-file\") pod \"limitador-limitador-54d6d988d5-nhksq\" (UID: \"704997ce-7891-4120-a9ca-a114f41ed771\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nhksq" Jul 22 16:16:43.461270 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.461105 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdpj2\" (UniqueName: \"kubernetes.io/projected/704997ce-7891-4120-a9ca-a114f41ed771-kube-api-access-gdpj2\") pod \"limitador-limitador-54d6d988d5-nhksq\" (UID: \"704997ce-7891-4120-a9ca-a114f41ed771\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nhksq" Jul 22 16:16:43.562388 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.562356 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdpj2\" (UniqueName: \"kubernetes.io/projected/704997ce-7891-4120-a9ca-a114f41ed771-kube-api-access-gdpj2\") pod \"limitador-limitador-54d6d988d5-nhksq\" (UID: \"704997ce-7891-4120-a9ca-a114f41ed771\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nhksq" Jul 22 16:16:43.562511 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.562461 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/704997ce-7891-4120-a9ca-a114f41ed771-config-file\") pod \"limitador-limitador-54d6d988d5-nhksq\" (UID: \"704997ce-7891-4120-a9ca-a114f41ed771\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nhksq" Jul 22 16:16:43.563086 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.563068 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/704997ce-7891-4120-a9ca-a114f41ed771-config-file\") pod \"limitador-limitador-54d6d988d5-nhksq\" (UID: \"704997ce-7891-4120-a9ca-a114f41ed771\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nhksq" Jul 22 16:16:43.571902 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.571882 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdpj2\" (UniqueName: \"kubernetes.io/projected/704997ce-7891-4120-a9ca-a114f41ed771-kube-api-access-gdpj2\") pod \"limitador-limitador-54d6d988d5-nhksq\" (UID: \"704997ce-7891-4120-a9ca-a114f41ed771\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nhksq" Jul 22 16:16:43.654794 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.654733 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nhksq" Jul 22 16:16:43.780477 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:43.780436 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:16:43.783145 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:16:43.783105 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704997ce_7891_4120_a9ca_a114f41ed771.slice/crio-ca1a078588efe2721212c35932d9214f0a44a5e7d5d8663bb32462f304485f1a WatchSource:0}: Error finding container ca1a078588efe2721212c35932d9214f0a44a5e7d5d8663bb32462f304485f1a: Status 404 returned error can't find the container with id ca1a078588efe2721212c35932d9214f0a44a5e7d5d8663bb32462f304485f1a Jul 22 16:16:44.424591 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:44.424508 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nhksq" event={"ID":"704997ce-7891-4120-a9ca-a114f41ed771","Type":"ContainerStarted","Data":"ca1a078588efe2721212c35932d9214f0a44a5e7d5d8663bb32462f304485f1a"} Jul 22 16:16:46.435956 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:46.435863 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nhksq" event={"ID":"704997ce-7891-4120-a9ca-a114f41ed771","Type":"ContainerStarted","Data":"94bd538dfeb4bed6cd723cf1f73c651ae65bfab917ddcedc52f87c69ddfd5b3d"} Jul 22 16:16:46.435956 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:46.435938 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nhksq" Jul 22 16:16:46.460457 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:46.460414 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-nhksq" podStartSLOduration=1.137343473 podStartE2EDuration="3.460401942s" totalImagesPullingTime="2.323058469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:16:46.459197085 +0000 UTC m=+571.601071160" watchObservedRunningTime="2026-07-22 16:16:46.460401942 +0000 UTC m=+571.602276029" Jul 22 16:16:57.441326 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:16:57.441296 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nhksq" Jul 22 16:17:15.851556 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:17:15.851528 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:17:15.854296 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:17:15.854278 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:17:15.855432 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:17:15.855413 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:17:15.858104 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:17:15.858087 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:17:56.314942 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:17:56.314909 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:18:23.148698 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:18:23.148662 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:18:31.555567 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:18:31.555532 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:18:57.780130 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:18:57.780095 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:19:02.352993 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:02.352960 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:19:10.526202 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.526131 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp"] Jul 22 16:19:10.528448 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.528426 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.531211 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.531192 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t84pc\"" Jul 22 16:19:10.532027 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.532004 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 16:19:10.532128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.532027 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 16:19:10.532128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.532004 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 22 16:19:10.546187 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.546131 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp"] Jul 22 16:19:10.606272 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.606241 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28240751-c131-445a-b78a-4ecf4dbcf4e3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.606399 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.606282 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28240751-c131-445a-b78a-4ecf4dbcf4e3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.606399 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.606361 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28240751-c131-445a-b78a-4ecf4dbcf4e3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.606542 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.606398 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpm2n\" (UniqueName: \"kubernetes.io/projected/28240751-c131-445a-b78a-4ecf4dbcf4e3-kube-api-access-jpm2n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.606542 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.606427 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28240751-c131-445a-b78a-4ecf4dbcf4e3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.606542 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.606487 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28240751-c131-445a-b78a-4ecf4dbcf4e3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.707184 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.707156 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28240751-c131-445a-b78a-4ecf4dbcf4e3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.707347 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.707192 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28240751-c131-445a-b78a-4ecf4dbcf4e3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.707347 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.707239 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28240751-c131-445a-b78a-4ecf4dbcf4e3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.707347 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.707277 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpm2n\" (UniqueName: \"kubernetes.io/projected/28240751-c131-445a-b78a-4ecf4dbcf4e3-kube-api-access-jpm2n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.707347 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.707302 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28240751-c131-445a-b78a-4ecf4dbcf4e3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.707347 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.707330 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28240751-c131-445a-b78a-4ecf4dbcf4e3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.707642 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.707619 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28240751-c131-445a-b78a-4ecf4dbcf4e3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.707730 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.707642 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28240751-c131-445a-b78a-4ecf4dbcf4e3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.707774 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.707730 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28240751-c131-445a-b78a-4ecf4dbcf4e3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.709465 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.709442 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28240751-c131-445a-b78a-4ecf4dbcf4e3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.709778 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.709761 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28240751-c131-445a-b78a-4ecf4dbcf4e3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.717823 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.717801 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpm2n\" (UniqueName: \"kubernetes.io/projected/28240751-c131-445a-b78a-4ecf4dbcf4e3-kube-api-access-jpm2n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-78dsp\" (UID: \"28240751-c131-445a-b78a-4ecf4dbcf4e3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.838561 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.838493 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:10.964052 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.964023 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp"] Jul 22 16:19:10.967369 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:19:10.967345 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28240751_c131_445a_b78a_4ecf4dbcf4e3.slice/crio-5950c582e3fb73bee64ad8350e2379bbaa62e7829320a8c7bf1c79ba6908f4dc WatchSource:0}: Error finding container 5950c582e3fb73bee64ad8350e2379bbaa62e7829320a8c7bf1c79ba6908f4dc: Status 404 returned error can't find the container with id 5950c582e3fb73bee64ad8350e2379bbaa62e7829320a8c7bf1c79ba6908f4dc Jul 22 16:19:10.968538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:10.968522 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:19:11.081668 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:11.081635 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" event={"ID":"28240751-c131-445a-b78a-4ecf4dbcf4e3","Type":"ContainerStarted","Data":"5950c582e3fb73bee64ad8350e2379bbaa62e7829320a8c7bf1c79ba6908f4dc"} Jul 22 16:19:13.333969 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:13.333938 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:19:14.097704 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:14.097651 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" event={"ID":"28240751-c131-445a-b78a-4ecf4dbcf4e3","Type":"ContainerStarted","Data":"b586e8a269c44414a3f84533a81f1304acd56cf1a14f6aaac37567acfa12f5d6"} Jul 22 16:19:14.119294 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:14.119247 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" podStartSLOduration=1.931795369 podStartE2EDuration="4.119231073s" totalImagesPullingTime="2.187435704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:19:14.117461645 +0000 UTC m=+719.259335713" watchObservedRunningTime="2026-07-22 16:19:14.119231073 +0000 UTC m=+719.261105149" Jul 22 16:19:20.839534 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:20.839497 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:20.840019 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:20.839560 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:20.851189 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:20.851161 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:21.140235 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:21.140159 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-78dsp" Jul 22 16:19:25.550295 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:19:25.550256 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:20:46.843557 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:20:46.843480 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:20:56.380081 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:20:56.380045 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:21:07.449261 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:21:07.449229 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:22:15.880148 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:22:15.880074 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:22:15.880531 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:22:15.880241 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:22:15.886469 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:22:15.886448 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:22:15.886605 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:22:15.886558 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:22:50.237458 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:22:50.237416 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:23:02.441227 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:23:02.441192 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:23:42.742124 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:23:42.742048 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:23:50.436202 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:23:50.436166 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:24:01.433019 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:24:01.432982 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:24:09.226859 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:24:09.226828 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:24:21.041250 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:24:21.041213 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:24:33.639130 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:24:33.639101 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:24:44.738723 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:24:44.738676 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:24:53.737967 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:24:53.737934 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:25:15.445317 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:25:15.445278 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:25:28.340229 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:25:28.340186 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:25:51.741223 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:25:51.741189 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:26:03.237710 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:26:03.237651 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:26:19.854494 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:26:19.854458 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:26:29.730556 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:26:29.730520 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:26:39.746207 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:26:39.746133 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:26:50.438312 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:26:50.438279 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:27:00.037133 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:27:00.037095 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:27:15.908271 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:27:15.908241 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:27:15.911085 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:27:15.911067 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:27:15.912103 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:27:15.912085 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:27:15.915020 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:27:15.915007 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:28:46.842824 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:28:46.842739 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:28:58.640249 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:28:58.640211 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:29:07.641136 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:29:07.641103 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:29:19.243007 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:29:19.242972 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:29:30.428180 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:29:30.428140 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:29:41.434529 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:29:41.434456 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:29:50.038084 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:29:50.038051 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:30:01.231564 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:30:01.231530 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:30:12.349095 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:30:12.349059 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:30:22.543573 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:30:22.543533 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:30:35.241120 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:30:35.241074 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:30:43.211514 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:30:43.211480 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:30:59.950492 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:30:59.950463 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:31:10.535371 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:31:10.535337 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:31:30.637185 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:31:30.637151 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:32:10.154491 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:32:10.154457 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:32:15.935511 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:32:15.935482 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:32:15.937601 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:32:15.937580 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:32:15.939208 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:32:15.939191 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:32:15.941344 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:32:15.941329 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:32:24.652515 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:32:24.652479 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:32:31.140145 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:32:31.140109 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:32:47.149186 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:32:47.149114 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:32:56.444783 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:32:56.444747 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:33:06.342662 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:33:06.342628 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:33:15.146452 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:33:15.146419 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:33:31.367653 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:33:31.367618 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:33:39.063532 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:33:39.063501 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:33:48.774659 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:33:48.774619 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:34:03.247592 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:34:03.247552 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:34:12.842905 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:34:12.842830 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:34:21.437484 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:34:21.437443 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:34:33.455753 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:34:33.455719 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:34:49.235510 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:34:49.235478 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:34:59.440658 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:34:59.440626 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:35:12.630513 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:12.630477 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:35:21.551169 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:21.551139 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:35:32.797704 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:32.797664 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:35:45.445335 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:45.445245 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:35:51.105556 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.105521 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j"] Jul 22 16:35:51.108860 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.108841 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.113553 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.113532 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d0a29ed9-openshift-default-dockercfg-2rvkg\"" Jul 22 16:35:51.129992 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.129968 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j"] Jul 22 16:35:51.284372 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.284325 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-token\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.284572 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.284390 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-workload-certs\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.284572 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.284411 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-podinfo\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.284572 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.284431 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-envoy\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.284572 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.284453 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad76726-6ab0-44cb-8470-077650a9a2bc-istiod-ca-cert\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.284572 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.284475 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-workload-socket\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.284862 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.284578 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6bxh\" (UniqueName: \"kubernetes.io/projected/7ad76726-6ab0-44cb-8470-077650a9a2bc-kube-api-access-h6bxh\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.284862 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.284621 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-credential-socket\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.284862 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.284647 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-data\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.386242 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.386128 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-token\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.386478 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.386280 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-workload-certs\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.386478 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.386313 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-podinfo\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.386478 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.386347 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-envoy\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.386478 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.386386 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad76726-6ab0-44cb-8470-077650a9a2bc-istiod-ca-cert\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.386478 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.386419 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-workload-socket\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.386478 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.386453 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6bxh\" (UniqueName: \"kubernetes.io/projected/7ad76726-6ab0-44cb-8470-077650a9a2bc-kube-api-access-h6bxh\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.386839 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.386481 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-credential-socket\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.386839 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.386510 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-data\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.386839 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.386778 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-workload-certs\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.386975 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.386899 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-data\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.387121 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.387081 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-workload-socket\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.387244 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.387126 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-credential-socket\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.387310 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.387252 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad76726-6ab0-44cb-8470-077650a9a2bc-istiod-ca-cert\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.388994 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.388971 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-envoy\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.389214 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.389187 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-podinfo\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.396446 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.396421 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-token\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.396606 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.396581 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6bxh\" (UniqueName: \"kubernetes.io/projected/7ad76726-6ab0-44cb-8470-077650a9a2bc-kube-api-access-h6bxh\") pod \"e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.419389 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.419358 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:51.555640 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.555562 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j"] Jul 22 16:35:51.557906 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:35:51.557876 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad76726_6ab0_44cb_8470_077650a9a2bc.slice/crio-07fab5ef2531489c50447d445b4647dc30f4ef54c0cad761db5672f80aeb79f8 WatchSource:0}: Error finding container 07fab5ef2531489c50447d445b4647dc30f4ef54c0cad761db5672f80aeb79f8: Status 404 returned error can't find the container with id 07fab5ef2531489c50447d445b4647dc30f4ef54c0cad761db5672f80aeb79f8 Jul 22 16:35:51.559987 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.559955 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:35:51.560082 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.560019 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:35:51.560082 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:51.560045 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:35:52.536698 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:52.536663 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" event={"ID":"7ad76726-6ab0-44cb-8470-077650a9a2bc","Type":"ContainerStarted","Data":"c5f6b4d4e78b4492a6e0fc407b209ef1a3d2f5f36c1845efa26c45b1312e25e0"} Jul 22 16:35:52.537078 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:52.536735 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" event={"ID":"7ad76726-6ab0-44cb-8470-077650a9a2bc","Type":"ContainerStarted","Data":"07fab5ef2531489c50447d445b4647dc30f4ef54c0cad761db5672f80aeb79f8"} Jul 22 16:35:52.557923 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:52.557871 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" podStartSLOduration=1.5578545350000002 podStartE2EDuration="1.557854535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:35:52.556652665 +0000 UTC m=+1717.698526753" watchObservedRunningTime="2026-07-22 16:35:52.557854535 +0000 UTC m=+1717.699728612" Jul 22 16:35:53.420191 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:53.420158 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:53.425206 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:53.425177 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:53.542967 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:53.542929 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:35:53.543917 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:35:53.543894 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:36:00.044991 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:00.044955 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:36:08.544371 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:08.544333 2567 patch_prober.go:28] interesting pod/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 22 16:36:08.544801 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:08.544402 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" podUID="7ad76726-6ab0-44cb-8470-077650a9a2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 22 16:36:08.614101 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:08.614069 2567 generic.go:356] "Generic (PLEG): container finished" podID="7ad76726-6ab0-44cb-8470-077650a9a2bc" containerID="c5f6b4d4e78b4492a6e0fc407b209ef1a3d2f5f36c1845efa26c45b1312e25e0" exitCode=0 Jul 22 16:36:08.614253 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:08.614144 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" event={"ID":"7ad76726-6ab0-44cb-8470-077650a9a2bc","Type":"ContainerDied","Data":"c5f6b4d4e78b4492a6e0fc407b209ef1a3d2f5f36c1845efa26c45b1312e25e0"} Jul 22 16:36:08.614564 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:08.614541 2567 scope.go:117] "RemoveContainer" containerID="c5f6b4d4e78b4492a6e0fc407b209ef1a3d2f5f36c1845efa26c45b1312e25e0" Jul 22 16:36:08.615568 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:08.615530 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:36:08.615670 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:08.615623 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:36:08.615742 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:08.615674 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:36:09.623780 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:09.623739 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" event={"ID":"7ad76726-6ab0-44cb-8470-077650a9a2bc","Type":"ContainerStarted","Data":"2829e2eb6487dd6016937847b18d12563e7b1d43bb1bbc842e8bbadc73f5d2e0"} Jul 22 16:36:10.419760 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:10.419718 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:36:10.420035 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:10.420002 2567 patch_prober.go:28] interesting pod/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 22 16:36:10.420241 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:10.420212 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" podUID="7ad76726-6ab0-44cb-8470-077650a9a2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 22 16:36:11.420033 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:11.419987 2567 patch_prober.go:28] interesting pod/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 22 16:36:11.420536 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:11.420064 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" podUID="7ad76726-6ab0-44cb-8470-077650a9a2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 22 16:36:12.420133 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:12.420088 2567 patch_prober.go:28] interesting pod/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 22 16:36:12.420626 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:12.420164 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" podUID="7ad76726-6ab0-44cb-8470-077650a9a2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 22 16:36:13.419955 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:13.419915 2567 patch_prober.go:28] interesting pod/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 22 16:36:13.420269 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:13.420235 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" podUID="7ad76726-6ab0-44cb-8470-077650a9a2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 22 16:36:14.425027 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:14.424994 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:36:14.425478 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:14.425391 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:36:14.426143 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:14.426122 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:36:33.443021 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:33.442980 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:36:34.765042 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:34.765005 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j"] Jul 22 16:36:34.765465 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:34.765307 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" podUID="7ad76726-6ab0-44cb-8470-077650a9a2bc" containerName="istio-proxy" containerID="cri-o://2829e2eb6487dd6016937847b18d12563e7b1d43bb1bbc842e8bbadc73f5d2e0" gracePeriod=30 Jul 22 16:36:35.783253 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:35.783211 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645"] Jul 22 16:36:35.786660 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:35.786641 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:35.788931 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:35.788912 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e47f5833-gw-openshift-default-dockercfg-lwv54\"" Jul 22 16:36:35.799597 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:35.799571 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645"] Jul 22 16:36:35.903601 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:35.903568 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-data\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:35.903804 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:35.903612 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-credential-socket\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:35.903804 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:35.903651 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-workload-certs\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:35.903804 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:35.903733 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-envoy\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:35.903804 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:35.903770 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56vq4\" (UniqueName: \"kubernetes.io/projected/5c3488b5-7420-49f3-a643-0d29bc24d021-kube-api-access-56vq4\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:35.903948 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:35.903843 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3488b5-7420-49f3-a643-0d29bc24d021-istiod-ca-cert\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:35.903948 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:35.903898 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-token\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:35.904012 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:35.903975 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-workload-socket\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:35.904012 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:35.903994 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-podinfo\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.005400 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.005366 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-data\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.005400 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.005404 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-credential-socket\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.005623 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.005423 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-workload-certs\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.005623 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.005536 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-envoy\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.005623 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.005584 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56vq4\" (UniqueName: \"kubernetes.io/projected/5c3488b5-7420-49f3-a643-0d29bc24d021-kube-api-access-56vq4\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.005623 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.005613 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3488b5-7420-49f3-a643-0d29bc24d021-istiod-ca-cert\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.005855 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.005671 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-token\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.005855 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.005778 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-workload-socket\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.005855 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.005800 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-workload-certs\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.005855 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.005803 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-podinfo\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.006050 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.005860 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-credential-socket\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.006050 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.005906 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-data\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.006162 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.006131 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-workload-socket\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.006381 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.006361 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3488b5-7420-49f3-a643-0d29bc24d021-istiod-ca-cert\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.007909 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.007883 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-envoy\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.008260 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.008240 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-podinfo\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.018060 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.018041 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-token\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.018246 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.018225 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56vq4\" (UniqueName: \"kubernetes.io/projected/5c3488b5-7420-49f3-a643-0d29bc24d021-kube-api-access-56vq4\") pod \"e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.096884 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.096792 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:36.242500 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.242474 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645"] Jul 22 16:36:36.246076 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:36:36.246035 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3488b5_7420_49f3_a643_0d29bc24d021.slice/crio-337380e1710ae62782528a77f9c9a7c637beb7b88334d98c2e7f07d29ceb0469 WatchSource:0}: Error finding container 337380e1710ae62782528a77f9c9a7c637beb7b88334d98c2e7f07d29ceb0469: Status 404 returned error can't find the container with id 337380e1710ae62782528a77f9c9a7c637beb7b88334d98c2e7f07d29ceb0469 Jul 22 16:36:36.248020 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.247984 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:36:36.248146 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.248062 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:36:36.248146 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.248106 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:36:36.750141 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.750103 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" event={"ID":"5c3488b5-7420-49f3-a643-0d29bc24d021","Type":"ContainerStarted","Data":"39acfe28bb48474b8104f23b774308542b8770e00529e422d820f31e2724bffc"} Jul 22 16:36:36.750327 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.750170 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" event={"ID":"5c3488b5-7420-49f3-a643-0d29bc24d021","Type":"ContainerStarted","Data":"337380e1710ae62782528a77f9c9a7c637beb7b88334d98c2e7f07d29ceb0469"} Jul 22 16:36:36.773147 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:36.773090 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" podStartSLOduration=1.7730721759999999 podStartE2EDuration="1.773072176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:36:36.771159748 +0000 UTC m=+1761.913033825" watchObservedRunningTime="2026-07-22 16:36:36.773072176 +0000 UTC m=+1761.914946253" Jul 22 16:36:37.097626 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:37.097592 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:37.102786 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:37.102761 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:37.754943 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:37.754911 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:37.755968 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:37.755948 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:40.027434 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.027359 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:36:40.144341 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.144306 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-envoy\") pod \"7ad76726-6ab0-44cb-8470-077650a9a2bc\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " Jul 22 16:36:40.144538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.144359 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-workload-socket\") pod \"7ad76726-6ab0-44cb-8470-077650a9a2bc\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " Jul 22 16:36:40.144538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.144414 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-token\") pod \"7ad76726-6ab0-44cb-8470-077650a9a2bc\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " Jul 22 16:36:40.144538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.144437 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad76726-6ab0-44cb-8470-077650a9a2bc-istiod-ca-cert\") pod \"7ad76726-6ab0-44cb-8470-077650a9a2bc\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " Jul 22 16:36:40.144538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.144476 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6bxh\" (UniqueName: \"kubernetes.io/projected/7ad76726-6ab0-44cb-8470-077650a9a2bc-kube-api-access-h6bxh\") pod \"7ad76726-6ab0-44cb-8470-077650a9a2bc\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " Jul 22 16:36:40.144538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.144500 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-data\") pod \"7ad76726-6ab0-44cb-8470-077650a9a2bc\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " Jul 22 16:36:40.144865 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.144558 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-podinfo\") pod \"7ad76726-6ab0-44cb-8470-077650a9a2bc\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " Jul 22 16:36:40.144865 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.144643 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-workload-certs\") pod \"7ad76726-6ab0-44cb-8470-077650a9a2bc\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " Jul 22 16:36:40.144865 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.144665 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-credential-socket\") pod \"7ad76726-6ab0-44cb-8470-077650a9a2bc\" (UID: \"7ad76726-6ab0-44cb-8470-077650a9a2bc\") " Jul 22 16:36:40.144865 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.144672 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7ad76726-6ab0-44cb-8470-077650a9a2bc" (UID: "7ad76726-6ab0-44cb-8470-077650a9a2bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:36:40.145068 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.144902 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad76726-6ab0-44cb-8470-077650a9a2bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7ad76726-6ab0-44cb-8470-077650a9a2bc" (UID: "7ad76726-6ab0-44cb-8470-077650a9a2bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:36:40.145068 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.145030 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7ad76726-6ab0-44cb-8470-077650a9a2bc" (UID: "7ad76726-6ab0-44cb-8470-077650a9a2bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:36:40.145234 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.145212 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:36:40.145296 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.145239 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:36:40.145296 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.145253 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad76726-6ab0-44cb-8470-077650a9a2bc-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:36:40.145573 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.145551 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7ad76726-6ab0-44cb-8470-077650a9a2bc" (UID: "7ad76726-6ab0-44cb-8470-077650a9a2bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:36:40.145830 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.145811 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "7ad76726-6ab0-44cb-8470-077650a9a2bc" (UID: "7ad76726-6ab0-44cb-8470-077650a9a2bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:36:40.146920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.146892 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7ad76726-6ab0-44cb-8470-077650a9a2bc" (UID: "7ad76726-6ab0-44cb-8470-077650a9a2bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:36:40.147023 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.146965 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7ad76726-6ab0-44cb-8470-077650a9a2bc" (UID: "7ad76726-6ab0-44cb-8470-077650a9a2bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 16:36:40.147023 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.147015 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "7ad76726-6ab0-44cb-8470-077650a9a2bc" (UID: "7ad76726-6ab0-44cb-8470-077650a9a2bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:36:40.147126 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.147090 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad76726-6ab0-44cb-8470-077650a9a2bc-kube-api-access-h6bxh" (OuterVolumeSpecName: "kube-api-access-h6bxh") pod "7ad76726-6ab0-44cb-8470-077650a9a2bc" (UID: "7ad76726-6ab0-44cb-8470-077650a9a2bc"). InnerVolumeSpecName "kube-api-access-h6bxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:36:40.246322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.246281 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6bxh\" (UniqueName: \"kubernetes.io/projected/7ad76726-6ab0-44cb-8470-077650a9a2bc-kube-api-access-h6bxh\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:36:40.246322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.246314 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:36:40.246322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.246329 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:36:40.246567 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.246337 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:36:40.246567 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.246348 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:36:40.246567 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.246357 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad76726-6ab0-44cb-8470-077650a9a2bc-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:36:40.770386 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.770350 2567 generic.go:356] "Generic (PLEG): container finished" podID="7ad76726-6ab0-44cb-8470-077650a9a2bc" containerID="2829e2eb6487dd6016937847b18d12563e7b1d43bb1bbc842e8bbadc73f5d2e0" exitCode=0 Jul 22 16:36:40.770575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.770399 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" Jul 22 16:36:40.770575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.770437 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" event={"ID":"7ad76726-6ab0-44cb-8470-077650a9a2bc","Type":"ContainerDied","Data":"2829e2eb6487dd6016937847b18d12563e7b1d43bb1bbc842e8bbadc73f5d2e0"} Jul 22 16:36:40.770575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.770496 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j" event={"ID":"7ad76726-6ab0-44cb-8470-077650a9a2bc","Type":"ContainerDied","Data":"07fab5ef2531489c50447d445b4647dc30f4ef54c0cad761db5672f80aeb79f8"} Jul 22 16:36:40.770575 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.770526 2567 scope.go:117] "RemoveContainer" containerID="2829e2eb6487dd6016937847b18d12563e7b1d43bb1bbc842e8bbadc73f5d2e0" Jul 22 16:36:40.782359 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.782334 2567 scope.go:117] "RemoveContainer" containerID="c5f6b4d4e78b4492a6e0fc407b209ef1a3d2f5f36c1845efa26c45b1312e25e0" Jul 22 16:36:40.790823 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.790795 2567 scope.go:117] "RemoveContainer" containerID="2829e2eb6487dd6016937847b18d12563e7b1d43bb1bbc842e8bbadc73f5d2e0" Jul 22 16:36:40.791158 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:36:40.791136 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2829e2eb6487dd6016937847b18d12563e7b1d43bb1bbc842e8bbadc73f5d2e0\": container with ID starting with 2829e2eb6487dd6016937847b18d12563e7b1d43bb1bbc842e8bbadc73f5d2e0 not found: ID does not exist" containerID="2829e2eb6487dd6016937847b18d12563e7b1d43bb1bbc842e8bbadc73f5d2e0" Jul 22 16:36:40.791198 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.791170 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2829e2eb6487dd6016937847b18d12563e7b1d43bb1bbc842e8bbadc73f5d2e0"} err="failed to get container status \"2829e2eb6487dd6016937847b18d12563e7b1d43bb1bbc842e8bbadc73f5d2e0\": rpc error: code = NotFound desc = could not find container \"2829e2eb6487dd6016937847b18d12563e7b1d43bb1bbc842e8bbadc73f5d2e0\": container with ID starting with 2829e2eb6487dd6016937847b18d12563e7b1d43bb1bbc842e8bbadc73f5d2e0 not found: ID does not exist" Jul 22 16:36:40.791198 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.791190 2567 scope.go:117] "RemoveContainer" containerID="c5f6b4d4e78b4492a6e0fc407b209ef1a3d2f5f36c1845efa26c45b1312e25e0" Jul 22 16:36:40.791471 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:36:40.791453 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f6b4d4e78b4492a6e0fc407b209ef1a3d2f5f36c1845efa26c45b1312e25e0\": container with ID starting with c5f6b4d4e78b4492a6e0fc407b209ef1a3d2f5f36c1845efa26c45b1312e25e0 not found: ID does not exist" containerID="c5f6b4d4e78b4492a6e0fc407b209ef1a3d2f5f36c1845efa26c45b1312e25e0" Jul 22 16:36:40.791510 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.791477 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f6b4d4e78b4492a6e0fc407b209ef1a3d2f5f36c1845efa26c45b1312e25e0"} err="failed to get container status \"c5f6b4d4e78b4492a6e0fc407b209ef1a3d2f5f36c1845efa26c45b1312e25e0\": rpc error: code = NotFound desc = could not find container \"c5f6b4d4e78b4492a6e0fc407b209ef1a3d2f5f36c1845efa26c45b1312e25e0\": container with ID starting with c5f6b4d4e78b4492a6e0fc407b209ef1a3d2f5f36c1845efa26c45b1312e25e0 not found: ID does not exist" Jul 22 16:36:40.796654 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.796625 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j"] Jul 22 16:36:40.800926 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:40.800890 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d0a29ed9-openshift-default-7c5b67b767-r7x2j"] Jul 22 16:36:41.984362 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:41.984327 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ad76726-6ab0-44cb-8470-077650a9a2bc" path="/var/lib/kubelet/pods/7ad76726-6ab0-44cb-8470-077650a9a2bc/volumes" Jul 22 16:36:44.433771 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:44.433730 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:36:52.756535 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:52.756499 2567 patch_prober.go:28] interesting pod/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Jul 22 16:36:52.757035 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:52.756570 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" podUID="5c3488b5-7420-49f3-a643-0d29bc24d021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Jul 22 16:36:52.835238 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:52.835202 2567 generic.go:356] "Generic (PLEG): container finished" podID="5c3488b5-7420-49f3-a643-0d29bc24d021" containerID="39acfe28bb48474b8104f23b774308542b8770e00529e422d820f31e2724bffc" exitCode=0 Jul 22 16:36:52.835399 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:52.835273 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" event={"ID":"5c3488b5-7420-49f3-a643-0d29bc24d021","Type":"ContainerDied","Data":"39acfe28bb48474b8104f23b774308542b8770e00529e422d820f31e2724bffc"} Jul 22 16:36:52.835705 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:52.835678 2567 scope.go:117] "RemoveContainer" containerID="39acfe28bb48474b8104f23b774308542b8770e00529e422d820f31e2724bffc" Jul 22 16:36:52.836649 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:52.836619 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:36:52.836761 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:52.836703 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:36:52.836761 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:52.836747 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:36:53.842171 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:53.842135 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" event={"ID":"5c3488b5-7420-49f3-a643-0d29bc24d021","Type":"ContainerStarted","Data":"9b648d03d6a33a0b12db912005685819241cf525615c2640291b2efad512759e"} Jul 22 16:36:54.097487 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:54.097397 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:54.097653 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:54.097635 2567 patch_prober.go:28] interesting pod/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Jul 22 16:36:54.097769 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:54.097734 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" podUID="5c3488b5-7420-49f3-a643-0d29bc24d021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Jul 22 16:36:55.097274 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:55.097230 2567 patch_prober.go:28] interesting pod/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Jul 22 16:36:55.097784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:55.097298 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" podUID="5c3488b5-7420-49f3-a643-0d29bc24d021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Jul 22 16:36:56.098264 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:56.098231 2567 patch_prober.go:28] interesting pod/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Jul 22 16:36:56.098752 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:56.098722 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" podUID="5c3488b5-7420-49f3-a643-0d29bc24d021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Jul 22 16:36:57.097652 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:57.097597 2567 patch_prober.go:28] interesting pod/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Jul 22 16:36:57.098068 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:57.098038 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" podUID="5c3488b5-7420-49f3-a643-0d29bc24d021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Jul 22 16:36:58.101919 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:58.101884 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:58.102368 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:58.102196 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:36:58.102946 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:36:58.102927 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:37:15.963444 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:15.963414 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:37:15.965678 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:15.965654 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:37:15.967535 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:15.967515 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:37:15.969423 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:15.969393 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:37:16.934020 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:16.933981 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:37:18.502638 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:18.502599 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645"] Jul 22 16:37:18.503070 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:18.502882 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" podUID="5c3488b5-7420-49f3-a643-0d29bc24d021" containerName="istio-proxy" containerID="cri-o://9b648d03d6a33a0b12db912005685819241cf525615c2640291b2efad512759e" gracePeriod=30 Jul 22 16:37:19.388480 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.388444 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv"] Jul 22 16:37:19.388878 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.388863 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad76726-6ab0-44cb-8470-077650a9a2bc" containerName="istio-proxy" Jul 22 16:37:19.388935 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.388880 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad76726-6ab0-44cb-8470-077650a9a2bc" containerName="istio-proxy" Jul 22 16:37:19.388935 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.388894 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad76726-6ab0-44cb-8470-077650a9a2bc" containerName="istio-proxy" Jul 22 16:37:19.388935 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.388900 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad76726-6ab0-44cb-8470-077650a9a2bc" containerName="istio-proxy" Jul 22 16:37:19.389043 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.388966 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad76726-6ab0-44cb-8470-077650a9a2bc" containerName="istio-proxy" Jul 22 16:37:19.389043 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.388974 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad76726-6ab0-44cb-8470-077650a9a2bc" containerName="istio-proxy" Jul 22 16:37:19.392092 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.392074 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.394369 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.394347 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4c7e4f77-openshift-default-dockercfg-lh5g4\"" Jul 22 16:37:19.414056 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.414029 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv"] Jul 22 16:37:19.527106 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.527068 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc256c7-9833-4b3e-8bf8-5354eb003904-istiod-ca-cert\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.527523 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.527120 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-data\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.527523 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.527188 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-credential-socket\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.527523 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.527217 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-workload-certs\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.527523 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.527243 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-podinfo\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.527523 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.527330 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-token\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.527523 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.527375 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5hwp\" (UniqueName: \"kubernetes.io/projected/cfc256c7-9833-4b3e-8bf8-5354eb003904-kube-api-access-h5hwp\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.527523 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.527433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-envoy\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.527523 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.527478 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-workload-socket\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.628080 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.628046 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-workload-certs\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.628080 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.628088 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-podinfo\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.628322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.628117 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-token\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.628322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.628145 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5hwp\" (UniqueName: \"kubernetes.io/projected/cfc256c7-9833-4b3e-8bf8-5354eb003904-kube-api-access-h5hwp\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.628322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.628188 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-envoy\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.628322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.628236 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-workload-socket\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.628525 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.628402 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc256c7-9833-4b3e-8bf8-5354eb003904-istiod-ca-cert\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.628525 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.628464 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-data\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.628525 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.628498 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-workload-certs\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.628684 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.628536 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-workload-socket\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.628684 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.628537 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-credential-socket\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.628861 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.628819 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-data\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.628941 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.628877 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-credential-socket\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.629194 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.629170 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc256c7-9833-4b3e-8bf8-5354eb003904-istiod-ca-cert\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.630572 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.630551 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-envoy\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.630797 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.630780 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-podinfo\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.636750 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.636727 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-token\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.636939 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.636923 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5hwp\" (UniqueName: \"kubernetes.io/projected/cfc256c7-9833-4b3e-8bf8-5354eb003904-kube-api-access-h5hwp\") pod \"e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.701860 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.701777 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:19.858550 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.858514 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv"] Jul 22 16:37:19.861479 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:37:19.861447 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc256c7_9833_4b3e_8bf8_5354eb003904.slice/crio-fa51f013752ad82e6304602722a0ba5c6fe20bdb96355bbc1829d3220dbc36b3 WatchSource:0}: Error finding container fa51f013752ad82e6304602722a0ba5c6fe20bdb96355bbc1829d3220dbc36b3: Status 404 returned error can't find the container with id fa51f013752ad82e6304602722a0ba5c6fe20bdb96355bbc1829d3220dbc36b3 Jul 22 16:37:19.863831 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.863785 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:37:19.863913 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.863880 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:37:19.863956 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.863922 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:37:19.984479 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.984400 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" event={"ID":"cfc256c7-9833-4b3e-8bf8-5354eb003904","Type":"ContainerStarted","Data":"ee0297bc698530f1d3390088d98a283a98930694712a6277dd009e449d6be68d"} Jul 22 16:37:19.984479 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:19.984450 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" event={"ID":"cfc256c7-9833-4b3e-8bf8-5354eb003904","Type":"ContainerStarted","Data":"fa51f013752ad82e6304602722a0ba5c6fe20bdb96355bbc1829d3220dbc36b3"} Jul 22 16:37:20.702401 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:20.702363 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:20.707093 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:20.707067 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:20.737304 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:20.737253 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" podStartSLOduration=1.737236592 podStartE2EDuration="1.737236592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:37:20.01367189 +0000 UTC m=+1805.155545971" watchObservedRunningTime="2026-07-22 16:37:20.737236592 +0000 UTC m=+1805.879110663" Jul 22 16:37:20.983744 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:20.983644 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:20.984683 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:20.984660 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:23.759427 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.759399 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:37:23.763835 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.763791 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-token\") pod \"5c3488b5-7420-49f3-a643-0d29bc24d021\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " Jul 22 16:37:23.763913 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.763854 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-credential-socket\") pod \"5c3488b5-7420-49f3-a643-0d29bc24d021\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " Jul 22 16:37:23.763970 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.763951 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-workload-socket\") pod \"5c3488b5-7420-49f3-a643-0d29bc24d021\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " Jul 22 16:37:23.764016 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.764003 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-workload-certs\") pod \"5c3488b5-7420-49f3-a643-0d29bc24d021\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " Jul 22 16:37:23.764061 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.764034 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-podinfo\") pod \"5c3488b5-7420-49f3-a643-0d29bc24d021\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " Jul 22 16:37:23.764113 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.764061 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56vq4\" (UniqueName: \"kubernetes.io/projected/5c3488b5-7420-49f3-a643-0d29bc24d021-kube-api-access-56vq4\") pod \"5c3488b5-7420-49f3-a643-0d29bc24d021\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " Jul 22 16:37:23.764113 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.764096 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5c3488b5-7420-49f3-a643-0d29bc24d021" (UID: "5c3488b5-7420-49f3-a643-0d29bc24d021"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:37:23.764207 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.764114 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-envoy\") pod \"5c3488b5-7420-49f3-a643-0d29bc24d021\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " Jul 22 16:37:23.764259 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.764206 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5c3488b5-7420-49f3-a643-0d29bc24d021" (UID: "5c3488b5-7420-49f3-a643-0d29bc24d021"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:37:23.764313 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.764269 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-data\") pod \"5c3488b5-7420-49f3-a643-0d29bc24d021\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " Jul 22 16:37:23.764313 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.764281 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5c3488b5-7420-49f3-a643-0d29bc24d021" (UID: "5c3488b5-7420-49f3-a643-0d29bc24d021"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:37:23.764420 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.764311 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3488b5-7420-49f3-a643-0d29bc24d021-istiod-ca-cert\") pod \"5c3488b5-7420-49f3-a643-0d29bc24d021\" (UID: \"5c3488b5-7420-49f3-a643-0d29bc24d021\") " Jul 22 16:37:23.764676 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.764613 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:37:23.764676 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.764635 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:37:23.764676 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.764651 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:37:23.764971 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.764752 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3488b5-7420-49f3-a643-0d29bc24d021-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5c3488b5-7420-49f3-a643-0d29bc24d021" (UID: "5c3488b5-7420-49f3-a643-0d29bc24d021"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:37:23.765177 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.765153 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-data" (OuterVolumeSpecName: "istio-data") pod "5c3488b5-7420-49f3-a643-0d29bc24d021" (UID: "5c3488b5-7420-49f3-a643-0d29bc24d021"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:37:23.766417 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.766390 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3488b5-7420-49f3-a643-0d29bc24d021-kube-api-access-56vq4" (OuterVolumeSpecName: "kube-api-access-56vq4") pod "5c3488b5-7420-49f3-a643-0d29bc24d021" (UID: "5c3488b5-7420-49f3-a643-0d29bc24d021"). InnerVolumeSpecName "kube-api-access-56vq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:37:23.766506 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.766480 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5c3488b5-7420-49f3-a643-0d29bc24d021" (UID: "5c3488b5-7420-49f3-a643-0d29bc24d021"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:37:23.766667 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.766646 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-token" (OuterVolumeSpecName: "istio-token") pod "5c3488b5-7420-49f3-a643-0d29bc24d021" (UID: "5c3488b5-7420-49f3-a643-0d29bc24d021"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:37:23.766794 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.766774 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5c3488b5-7420-49f3-a643-0d29bc24d021" (UID: "5c3488b5-7420-49f3-a643-0d29bc24d021"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 16:37:23.865391 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.865345 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:37:23.865391 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.865378 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:37:23.865391 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.865389 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3488b5-7420-49f3-a643-0d29bc24d021-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:37:23.865391 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.865399 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:37:23.865658 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.865407 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3488b5-7420-49f3-a643-0d29bc24d021-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:37:23.865658 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.865415 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56vq4\" (UniqueName: \"kubernetes.io/projected/5c3488b5-7420-49f3-a643-0d29bc24d021-kube-api-access-56vq4\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:37:23.998926 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.998884 2567 generic.go:356] "Generic (PLEG): container finished" podID="5c3488b5-7420-49f3-a643-0d29bc24d021" containerID="9b648d03d6a33a0b12db912005685819241cf525615c2640291b2efad512759e" exitCode=0 Jul 22 16:37:23.999058 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.998944 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" Jul 22 16:37:23.999058 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.998948 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" event={"ID":"5c3488b5-7420-49f3-a643-0d29bc24d021","Type":"ContainerDied","Data":"9b648d03d6a33a0b12db912005685819241cf525615c2640291b2efad512759e"} Jul 22 16:37:23.999058 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.998993 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645" event={"ID":"5c3488b5-7420-49f3-a643-0d29bc24d021","Type":"ContainerDied","Data":"337380e1710ae62782528a77f9c9a7c637beb7b88334d98c2e7f07d29ceb0469"} Jul 22 16:37:23.999058 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:23.999018 2567 scope.go:117] "RemoveContainer" containerID="9b648d03d6a33a0b12db912005685819241cf525615c2640291b2efad512759e" Jul 22 16:37:24.007609 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:24.007589 2567 scope.go:117] "RemoveContainer" containerID="39acfe28bb48474b8104f23b774308542b8770e00529e422d820f31e2724bffc" Jul 22 16:37:24.015390 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:24.015343 2567 scope.go:117] "RemoveContainer" containerID="9b648d03d6a33a0b12db912005685819241cf525615c2640291b2efad512759e" Jul 22 16:37:24.015778 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:37:24.015664 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b648d03d6a33a0b12db912005685819241cf525615c2640291b2efad512759e\": container with ID starting with 9b648d03d6a33a0b12db912005685819241cf525615c2640291b2efad512759e not found: ID does not exist" containerID="9b648d03d6a33a0b12db912005685819241cf525615c2640291b2efad512759e" Jul 22 16:37:24.015778 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:24.015718 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b648d03d6a33a0b12db912005685819241cf525615c2640291b2efad512759e"} err="failed to get container status \"9b648d03d6a33a0b12db912005685819241cf525615c2640291b2efad512759e\": rpc error: code = NotFound desc = could not find container \"9b648d03d6a33a0b12db912005685819241cf525615c2640291b2efad512759e\": container with ID starting with 9b648d03d6a33a0b12db912005685819241cf525615c2640291b2efad512759e not found: ID does not exist" Jul 22 16:37:24.015778 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:24.015746 2567 scope.go:117] "RemoveContainer" containerID="39acfe28bb48474b8104f23b774308542b8770e00529e422d820f31e2724bffc" Jul 22 16:37:24.016170 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:37:24.016143 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39acfe28bb48474b8104f23b774308542b8770e00529e422d820f31e2724bffc\": container with ID starting with 39acfe28bb48474b8104f23b774308542b8770e00529e422d820f31e2724bffc not found: ID does not exist" containerID="39acfe28bb48474b8104f23b774308542b8770e00529e422d820f31e2724bffc" Jul 22 16:37:24.016258 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:24.016178 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39acfe28bb48474b8104f23b774308542b8770e00529e422d820f31e2724bffc"} err="failed to get container status \"39acfe28bb48474b8104f23b774308542b8770e00529e422d820f31e2724bffc\": rpc error: code = NotFound desc = could not find container \"39acfe28bb48474b8104f23b774308542b8770e00529e422d820f31e2724bffc\": container with ID starting with 39acfe28bb48474b8104f23b774308542b8770e00529e422d820f31e2724bffc not found: ID does not exist" Jul 22 16:37:24.018078 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:24.018060 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645"] Jul 22 16:37:24.022857 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:24.022833 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e47f5833-gw-openshift-default-6cbb75cb95-kh645"] Jul 22 16:37:25.984168 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:25.984132 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c3488b5-7420-49f3-a643-0d29bc24d021" path="/var/lib/kubelet/pods/5c3488b5-7420-49f3-a643-0d29bc24d021/volumes" Jul 22 16:37:28.232306 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:28.232272 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:37:36.018193 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:36.017383 2567 patch_prober.go:28] interesting pod/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": read tcp 10.132.0.2:35100->10.132.0.34:15021: read: connection reset by peer" start-of-body= Jul 22 16:37:36.018193 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:36.017472 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" podUID="cfc256c7-9833-4b3e-8bf8-5354eb003904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": read tcp 10.132.0.2:35100->10.132.0.34:15021: read: connection reset by peer" Jul 22 16:37:36.062195 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:36.062164 2567 generic.go:356] "Generic (PLEG): container finished" podID="cfc256c7-9833-4b3e-8bf8-5354eb003904" containerID="ee0297bc698530f1d3390088d98a283a98930694712a6277dd009e449d6be68d" exitCode=0 Jul 22 16:37:36.062353 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:36.062232 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" event={"ID":"cfc256c7-9833-4b3e-8bf8-5354eb003904","Type":"ContainerDied","Data":"ee0297bc698530f1d3390088d98a283a98930694712a6277dd009e449d6be68d"} Jul 22 16:37:36.062666 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:36.062649 2567 scope.go:117] "RemoveContainer" containerID="ee0297bc698530f1d3390088d98a283a98930694712a6277dd009e449d6be68d" Jul 22 16:37:36.063793 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:36.063765 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:37:36.063906 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:36.063846 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:37:36.063906 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:36.063894 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:37:37.068653 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:37.068614 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" event={"ID":"cfc256c7-9833-4b3e-8bf8-5354eb003904","Type":"ContainerStarted","Data":"6a449e1c1e5e0f7662529442f245b4ef4c9801aeae2648f2f69b68c30d4bb870"} Jul 22 16:37:37.702900 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:37.702853 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:37.703174 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:37.703145 2567 patch_prober.go:28] interesting pod/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 22 16:37:37.703255 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:37.703214 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" podUID="cfc256c7-9833-4b3e-8bf8-5354eb003904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 22 16:37:38.702716 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:38.702653 2567 patch_prober.go:28] interesting pod/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 22 16:37:38.703341 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:38.703304 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" podUID="cfc256c7-9833-4b3e-8bf8-5354eb003904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 22 16:37:39.703191 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:39.703154 2567 patch_prober.go:28] interesting pod/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 22 16:37:39.703816 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:39.703779 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" podUID="cfc256c7-9833-4b3e-8bf8-5354eb003904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 22 16:37:40.702255 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:40.702212 2567 patch_prober.go:28] interesting pod/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 22 16:37:40.702557 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:40.702525 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" podUID="cfc256c7-9833-4b3e-8bf8-5354eb003904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 22 16:37:41.706355 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:41.706324 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:41.706880 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:41.706707 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:37:41.707286 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:37:41.707268 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:38:01.235458 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:01.235419 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:38:03.643208 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:03.643172 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv"] Jul 22 16:38:03.643631 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:03.643460 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" podUID="cfc256c7-9833-4b3e-8bf8-5354eb003904" containerName="istio-proxy" containerID="cri-o://6a449e1c1e5e0f7662529442f245b4ef4c9801aeae2648f2f69b68c30d4bb870" gracePeriod=30 Jul 22 16:38:08.895902 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.895873 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:38:08.980368 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.980271 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc256c7-9833-4b3e-8bf8-5354eb003904-istiod-ca-cert\") pod \"cfc256c7-9833-4b3e-8bf8-5354eb003904\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " Jul 22 16:38:08.980368 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.980326 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-credential-socket\") pod \"cfc256c7-9833-4b3e-8bf8-5354eb003904\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " Jul 22 16:38:08.980603 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.980422 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-data\") pod \"cfc256c7-9833-4b3e-8bf8-5354eb003904\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " Jul 22 16:38:08.980603 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.980455 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-podinfo\") pod \"cfc256c7-9833-4b3e-8bf8-5354eb003904\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " Jul 22 16:38:08.980603 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.980504 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-envoy\") pod \"cfc256c7-9833-4b3e-8bf8-5354eb003904\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " Jul 22 16:38:08.980603 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.980522 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-workload-certs\") pod \"cfc256c7-9833-4b3e-8bf8-5354eb003904\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " Jul 22 16:38:08.980603 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.980526 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cfc256c7-9833-4b3e-8bf8-5354eb003904" (UID: "cfc256c7-9833-4b3e-8bf8-5354eb003904"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:38:08.980603 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.980551 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5hwp\" (UniqueName: \"kubernetes.io/projected/cfc256c7-9833-4b3e-8bf8-5354eb003904-kube-api-access-h5hwp\") pod \"cfc256c7-9833-4b3e-8bf8-5354eb003904\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " Jul 22 16:38:08.980603 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.980592 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-token\") pod \"cfc256c7-9833-4b3e-8bf8-5354eb003904\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " Jul 22 16:38:08.980977 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.980678 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-workload-socket\") pod \"cfc256c7-9833-4b3e-8bf8-5354eb003904\" (UID: \"cfc256c7-9833-4b3e-8bf8-5354eb003904\") " Jul 22 16:38:08.980977 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.980739 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc256c7-9833-4b3e-8bf8-5354eb003904-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cfc256c7-9833-4b3e-8bf8-5354eb003904" (UID: "cfc256c7-9833-4b3e-8bf8-5354eb003904"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:38:08.981086 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.980988 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc256c7-9833-4b3e-8bf8-5354eb003904-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:38:08.981086 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.981009 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:38:08.981086 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.981013 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cfc256c7-9833-4b3e-8bf8-5354eb003904" (UID: "cfc256c7-9833-4b3e-8bf8-5354eb003904"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:38:08.981086 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.981072 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cfc256c7-9833-4b3e-8bf8-5354eb003904" (UID: "cfc256c7-9833-4b3e-8bf8-5354eb003904"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:38:08.981239 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.981166 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-data" (OuterVolumeSpecName: "istio-data") pod "cfc256c7-9833-4b3e-8bf8-5354eb003904" (UID: "cfc256c7-9833-4b3e-8bf8-5354eb003904"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:38:08.982795 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.982765 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cfc256c7-9833-4b3e-8bf8-5354eb003904" (UID: "cfc256c7-9833-4b3e-8bf8-5354eb003904"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 16:38:08.982936 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.982917 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cfc256c7-9833-4b3e-8bf8-5354eb003904" (UID: "cfc256c7-9833-4b3e-8bf8-5354eb003904"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:38:08.982987 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.982944 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc256c7-9833-4b3e-8bf8-5354eb003904-kube-api-access-h5hwp" (OuterVolumeSpecName: "kube-api-access-h5hwp") pod "cfc256c7-9833-4b3e-8bf8-5354eb003904" (UID: "cfc256c7-9833-4b3e-8bf8-5354eb003904"). InnerVolumeSpecName "kube-api-access-h5hwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:38:08.983144 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:08.983127 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-token" (OuterVolumeSpecName: "istio-token") pod "cfc256c7-9833-4b3e-8bf8-5354eb003904" (UID: "cfc256c7-9833-4b3e-8bf8-5354eb003904"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:38:09.081658 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.081624 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:38:09.081658 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.081654 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:38:09.081658 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.081663 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:38:09.081929 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.081672 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:38:09.081929 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.081681 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc256c7-9833-4b3e-8bf8-5354eb003904-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:38:09.081929 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.081703 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5hwp\" (UniqueName: \"kubernetes.io/projected/cfc256c7-9833-4b3e-8bf8-5354eb003904-kube-api-access-h5hwp\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:38:09.081929 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.081714 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc256c7-9833-4b3e-8bf8-5354eb003904-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:38:09.223498 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.223464 2567 generic.go:356] "Generic (PLEG): container finished" podID="cfc256c7-9833-4b3e-8bf8-5354eb003904" containerID="6a449e1c1e5e0f7662529442f245b4ef4c9801aeae2648f2f69b68c30d4bb870" exitCode=0 Jul 22 16:38:09.223664 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.223524 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" Jul 22 16:38:09.223664 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.223555 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" event={"ID":"cfc256c7-9833-4b3e-8bf8-5354eb003904","Type":"ContainerDied","Data":"6a449e1c1e5e0f7662529442f245b4ef4c9801aeae2648f2f69b68c30d4bb870"} Jul 22 16:38:09.223664 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.223647 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv" event={"ID":"cfc256c7-9833-4b3e-8bf8-5354eb003904","Type":"ContainerDied","Data":"fa51f013752ad82e6304602722a0ba5c6fe20bdb96355bbc1829d3220dbc36b3"} Jul 22 16:38:09.223819 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.223675 2567 scope.go:117] "RemoveContainer" containerID="6a449e1c1e5e0f7662529442f245b4ef4c9801aeae2648f2f69b68c30d4bb870" Jul 22 16:38:09.233149 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.233131 2567 scope.go:117] "RemoveContainer" containerID="ee0297bc698530f1d3390088d98a283a98930694712a6277dd009e449d6be68d" Jul 22 16:38:09.241523 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.241502 2567 scope.go:117] "RemoveContainer" containerID="6a449e1c1e5e0f7662529442f245b4ef4c9801aeae2648f2f69b68c30d4bb870" Jul 22 16:38:09.241896 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:38:09.241877 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a449e1c1e5e0f7662529442f245b4ef4c9801aeae2648f2f69b68c30d4bb870\": container with ID starting with 6a449e1c1e5e0f7662529442f245b4ef4c9801aeae2648f2f69b68c30d4bb870 not found: ID does not exist" containerID="6a449e1c1e5e0f7662529442f245b4ef4c9801aeae2648f2f69b68c30d4bb870" Jul 22 16:38:09.241947 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.241906 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a449e1c1e5e0f7662529442f245b4ef4c9801aeae2648f2f69b68c30d4bb870"} err="failed to get container status \"6a449e1c1e5e0f7662529442f245b4ef4c9801aeae2648f2f69b68c30d4bb870\": rpc error: code = NotFound desc = could not find container \"6a449e1c1e5e0f7662529442f245b4ef4c9801aeae2648f2f69b68c30d4bb870\": container with ID starting with 6a449e1c1e5e0f7662529442f245b4ef4c9801aeae2648f2f69b68c30d4bb870 not found: ID does not exist" Jul 22 16:38:09.241947 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.241926 2567 scope.go:117] "RemoveContainer" containerID="ee0297bc698530f1d3390088d98a283a98930694712a6277dd009e449d6be68d" Jul 22 16:38:09.242179 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:38:09.242160 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee0297bc698530f1d3390088d98a283a98930694712a6277dd009e449d6be68d\": container with ID starting with ee0297bc698530f1d3390088d98a283a98930694712a6277dd009e449d6be68d not found: ID does not exist" containerID="ee0297bc698530f1d3390088d98a283a98930694712a6277dd009e449d6be68d" Jul 22 16:38:09.242219 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.242185 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0297bc698530f1d3390088d98a283a98930694712a6277dd009e449d6be68d"} err="failed to get container status \"ee0297bc698530f1d3390088d98a283a98930694712a6277dd009e449d6be68d\": rpc error: code = NotFound desc = could not find container \"ee0297bc698530f1d3390088d98a283a98930694712a6277dd009e449d6be68d\": container with ID starting with ee0297bc698530f1d3390088d98a283a98930694712a6277dd009e449d6be68d not found: ID does not exist" Jul 22 16:38:09.245948 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.245922 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv"] Jul 22 16:38:09.249931 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.249908 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4c7e4f77-openshift-default-865887f985-5t7tv"] Jul 22 16:38:09.983913 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:09.983881 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfc256c7-9833-4b3e-8bf8-5354eb003904" path="/var/lib/kubelet/pods/cfc256c7-9833-4b3e-8bf8-5354eb003904/volumes" Jul 22 16:38:16.037786 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:16.037759 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:38:48.833405 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:38:48.833314 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:40:16.733626 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:40:16.733549 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:40:27.430603 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:40:27.430562 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:40:38.630209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:40:38.630169 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:40:56.733558 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:40:56.733518 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:41:33.144616 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:41:33.144574 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:41:53.331543 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:41:53.331506 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:42:07.544266 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:42:07.544217 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:42:15.992148 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:42:15.992117 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:42:15.994788 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:42:15.994760 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:42:15.996400 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:42:15.996382 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:42:15.998842 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:42:15.998818 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:42:25.354962 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:42:25.354915 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:42:34.229466 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:42:34.229427 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:43:25.846630 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:43:25.846540 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:43:37.831898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:43:37.831858 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:44:26.236010 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.235966 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt"] Jul 22 16:44:26.236557 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.236420 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc256c7-9833-4b3e-8bf8-5354eb003904" containerName="istio-proxy" Jul 22 16:44:26.236557 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.236433 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc256c7-9833-4b3e-8bf8-5354eb003904" containerName="istio-proxy" Jul 22 16:44:26.236557 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.236449 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c3488b5-7420-49f3-a643-0d29bc24d021" containerName="istio-proxy" Jul 22 16:44:26.236557 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.236454 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c3488b5-7420-49f3-a643-0d29bc24d021" containerName="istio-proxy" Jul 22 16:44:26.236557 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.236461 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c3488b5-7420-49f3-a643-0d29bc24d021" containerName="istio-proxy" Jul 22 16:44:26.236557 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.236469 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c3488b5-7420-49f3-a643-0d29bc24d021" containerName="istio-proxy" Jul 22 16:44:26.236557 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.236540 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfc256c7-9833-4b3e-8bf8-5354eb003904" containerName="istio-proxy" Jul 22 16:44:26.236557 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.236557 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c3488b5-7420-49f3-a643-0d29bc24d021" containerName="istio-proxy" Jul 22 16:44:26.236860 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.236566 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c3488b5-7420-49f3-a643-0d29bc24d021" containerName="istio-proxy" Jul 22 16:44:26.239631 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.239608 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.242468 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.242446 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-059dda-openshift-default-dockercfg-v7gzd\"" Jul 22 16:44:26.253838 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.253805 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt"] Jul 22 16:44:26.312486 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.312439 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-token\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.312486 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.312491 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istiod-ca-cert\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.312715 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.312548 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-data\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.312715 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.312584 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-podinfo\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.312715 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.312611 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-credential-socket\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.312715 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.312628 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-envoy\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.312890 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.312704 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7p2s\" (UniqueName: \"kubernetes.io/projected/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-kube-api-access-m7p2s\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.312890 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.312776 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-workload-socket\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.312890 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.312810 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-workload-certs\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.413307 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.413260 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-credential-socket\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.413307 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.413305 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-envoy\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.413555 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.413343 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7p2s\" (UniqueName: \"kubernetes.io/projected/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-kube-api-access-m7p2s\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.413555 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.413370 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-workload-socket\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.413555 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.413391 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-workload-certs\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.413555 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.413415 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-token\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.413555 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.413434 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istiod-ca-cert\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.413555 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.413473 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-data\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.413555 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.413541 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-podinfo\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.413943 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.413799 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-credential-socket\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.413943 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.413924 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-workload-certs\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.414025 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.413936 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-workload-socket\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.414025 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.413969 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-data\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.414338 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.414319 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istiod-ca-cert\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.415821 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.415799 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-envoy\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.416078 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.416063 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-podinfo\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.424038 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.424007 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-token\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.424339 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.424317 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7p2s\" (UniqueName: \"kubernetes.io/projected/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-kube-api-access-m7p2s\") pod \"e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.438434 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.438396 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:44:26.549815 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.549771 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:26.696797 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.696758 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt"] Jul 22 16:44:26.699277 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:44:26.699248 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode925ef1f_2c5f_4766_a9c7_cc0868a4f961.slice/crio-f9e6601634168677c27359a5b063944a088559d32058d51771400550a7c9feae WatchSource:0}: Error finding container f9e6601634168677c27359a5b063944a088559d32058d51771400550a7c9feae: Status 404 returned error can't find the container with id f9e6601634168677c27359a5b063944a088559d32058d51771400550a7c9feae Jul 22 16:44:26.701393 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.701359 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:44:26.701475 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.701424 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:44:26.701475 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.701454 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:44:26.970604 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.970512 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" event={"ID":"e925ef1f-2c5f-4766-a9c7-cc0868a4f961","Type":"ContainerStarted","Data":"3079734f71eae5bcf88a74e4e216fb74aa07132809455687c0c522926f4c6148"} Jul 22 16:44:26.970771 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.970613 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" event={"ID":"e925ef1f-2c5f-4766-a9c7-cc0868a4f961","Type":"ContainerStarted","Data":"f9e6601634168677c27359a5b063944a088559d32058d51771400550a7c9feae"} Jul 22 16:44:26.992960 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:26.992904 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" podStartSLOduration=0.992887119 podStartE2EDuration="992.887119ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:44:26.99058787 +0000 UTC m=+2232.132461973" watchObservedRunningTime="2026-07-22 16:44:26.992887119 +0000 UTC m=+2232.134761191" Jul 22 16:44:27.550761 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:27.550715 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:27.555638 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:27.555612 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:27.975409 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:27.975321 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:27.976388 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:27.976364 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:39.246472 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:39.246371 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:44:40.041506 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:40.041459 2567 generic.go:356] "Generic (PLEG): container finished" podID="e925ef1f-2c5f-4766-a9c7-cc0868a4f961" containerID="3079734f71eae5bcf88a74e4e216fb74aa07132809455687c0c522926f4c6148" exitCode=0 Jul 22 16:44:40.041731 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:40.041535 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" event={"ID":"e925ef1f-2c5f-4766-a9c7-cc0868a4f961","Type":"ContainerDied","Data":"3079734f71eae5bcf88a74e4e216fb74aa07132809455687c0c522926f4c6148"} Jul 22 16:44:40.042087 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:40.042070 2567 scope.go:117] "RemoveContainer" containerID="3079734f71eae5bcf88a74e4e216fb74aa07132809455687c0c522926f4c6148" Jul 22 16:44:40.043114 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:40.043084 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:44:40.043174 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:40.043152 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:44:40.043209 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:40.043189 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:44:41.049649 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:41.049608 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" event={"ID":"e925ef1f-2c5f-4766-a9c7-cc0868a4f961","Type":"ContainerStarted","Data":"95e96fbd9ce2351b7844b56e746d45e51a1eea212e1af511dac2b9f700477ade"} Jul 22 16:44:41.550513 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:41.550472 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:41.551680 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:41.551649 2567 patch_prober.go:28] interesting pod/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 22 16:44:41.551816 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:41.551726 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" podUID="e925ef1f-2c5f-4766-a9c7-cc0868a4f961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 22 16:44:42.550994 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:42.550952 2567 patch_prober.go:28] interesting pod/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 22 16:44:42.551387 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:42.551012 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" podUID="e925ef1f-2c5f-4766-a9c7-cc0868a4f961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 22 16:44:42.975810 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:42.975720 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:43.550933 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:43.550895 2567 patch_prober.go:28] interesting pod/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 22 16:44:43.551111 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:43.550972 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" podUID="e925ef1f-2c5f-4766-a9c7-cc0868a4f961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 22 16:44:44.550772 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:44.550731 2567 patch_prober.go:28] interesting pod/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 22 16:44:44.550962 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:44.550805 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" podUID="e925ef1f-2c5f-4766-a9c7-cc0868a4f961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 22 16:44:45.554183 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:45.554147 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:45.555040 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:45.555018 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:44:50.638789 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:44:50.638750 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:45:00.150351 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.150311 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745645-v8mq7"] Jul 22 16:45:00.150865 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.150808 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc256c7-9833-4b3e-8bf8-5354eb003904" containerName="istio-proxy" Jul 22 16:45:00.150865 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.150824 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc256c7-9833-4b3e-8bf8-5354eb003904" containerName="istio-proxy" Jul 22 16:45:00.150938 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.150898 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfc256c7-9833-4b3e-8bf8-5354eb003904" containerName="istio-proxy" Jul 22 16:45:00.153863 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.153844 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-059dda-29745645-6rmsq"] Jul 22 16:45:00.153977 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.153956 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745645-v8mq7" Jul 22 16:45:00.156089 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.156066 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfdj\"" Jul 22 16:45:00.157087 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.157068 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-059dda-29745645-6rmsq" Jul 22 16:45:00.163821 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.163792 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-059dda-29745645-6rmsq"] Jul 22 16:45:00.165741 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.165718 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745645-v8mq7"] Jul 22 16:45:00.169597 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.169577 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745645-v8mq7" Jul 22 16:45:00.176343 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.176325 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-059dda-29745645-6rmsq" Jul 22 16:45:00.326800 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.326775 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-059dda-29745645-6rmsq"] Jul 22 16:45:00.328901 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:45:00.328872 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9dec27_5e57_41a7_80f1_1e7ff8b70fc0.slice/crio-af7f24a8dc5898298b4e224378efd6a3637c6257a2a3d660ab23b72673c87b43 WatchSource:0}: Error finding container af7f24a8dc5898298b4e224378efd6a3637c6257a2a3d660ab23b72673c87b43: Status 404 returned error can't find the container with id af7f24a8dc5898298b4e224378efd6a3637c6257a2a3d660ab23b72673c87b43 Jul 22 16:45:00.331438 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.331420 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:45:00.345836 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:00.345779 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745645-v8mq7"] Jul 22 16:45:00.347826 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:45:00.347803 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed74baf_4f05_465a_baab_ff90f7e2bde5.slice/crio-e3cf6afd9e326ca2eeb2bad8d7912e8d2f24e872676640f14961248b85fb35d7 WatchSource:0}: Error finding container e3cf6afd9e326ca2eeb2bad8d7912e8d2f24e872676640f14961248b85fb35d7: Status 404 returned error can't find the container with id e3cf6afd9e326ca2eeb2bad8d7912e8d2f24e872676640f14961248b85fb35d7 Jul 22 16:45:01.148728 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:01.148673 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745645-v8mq7" event={"ID":"8ed74baf-4f05-465a-baab-ff90f7e2bde5","Type":"ContainerStarted","Data":"e3cf6afd9e326ca2eeb2bad8d7912e8d2f24e872676640f14961248b85fb35d7"} Jul 22 16:45:01.149832 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:01.149805 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-059dda-29745645-6rmsq" event={"ID":"7b9dec27-5e57-41a7-80f1-1e7ff8b70fc0","Type":"ContainerStarted","Data":"af7f24a8dc5898298b4e224378efd6a3637c6257a2a3d660ab23b72673c87b43"} Jul 22 16:45:03.164228 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:03.164188 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-059dda-29745645-6rmsq" event={"ID":"7b9dec27-5e57-41a7-80f1-1e7ff8b70fc0","Type":"ContainerStarted","Data":"00e7ca1a1a1605c1bc19fe1c8967039c7cce8850cf7e5de1631d9c847d628afd"} Jul 22 16:45:03.166001 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:03.165975 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745645-v8mq7" event={"ID":"8ed74baf-4f05-465a-baab-ff90f7e2bde5","Type":"ContainerStarted","Data":"8980bf0df36eebd412c5c680132a6e265bacce8507f6aadb52a7f854bdd5630b"} Jul 22 16:45:03.185978 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:03.185880 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-059dda-29745645-6rmsq" podStartSLOduration=0.851773643 podStartE2EDuration="3.185862554s" totalImagesPullingTime="2.334088911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:45:03.183926063 +0000 UTC m=+2268.325800144" watchObservedRunningTime="2026-07-22 16:45:03.185862554 +0000 UTC m=+2268.327736632" Jul 22 16:45:03.201444 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:03.201391 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745645-v8mq7" podStartSLOduration=0.890155151 podStartE2EDuration="3.201375005s" totalImagesPullingTime="2.311219854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:45:03.199035695 +0000 UTC m=+2268.340909777" watchObservedRunningTime="2026-07-22 16:45:03.201375005 +0000 UTC m=+2268.343249082" Jul 22 16:45:04.172835 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:04.172790 2567 generic.go:356] "Generic (PLEG): container finished" podID="7b9dec27-5e57-41a7-80f1-1e7ff8b70fc0" containerID="00e7ca1a1a1605c1bc19fe1c8967039c7cce8850cf7e5de1631d9c847d628afd" exitCode=0 Jul 22 16:45:04.173281 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:04.172881 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-059dda-29745645-6rmsq" event={"ID":"7b9dec27-5e57-41a7-80f1-1e7ff8b70fc0","Type":"ContainerDied","Data":"00e7ca1a1a1605c1bc19fe1c8967039c7cce8850cf7e5de1631d9c847d628afd"} Jul 22 16:45:04.174370 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:04.174346 2567 generic.go:356] "Generic (PLEG): container finished" podID="8ed74baf-4f05-465a-baab-ff90f7e2bde5" containerID="8980bf0df36eebd412c5c680132a6e265bacce8507f6aadb52a7f854bdd5630b" exitCode=0 Jul 22 16:45:04.174478 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:04.174419 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745645-v8mq7" event={"ID":"8ed74baf-4f05-465a-baab-ff90f7e2bde5","Type":"ContainerDied","Data":"8980bf0df36eebd412c5c680132a6e265bacce8507f6aadb52a7f854bdd5630b"} Jul 22 16:45:04.933226 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:04.933191 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:45:05.311848 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:05.311818 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-059dda-29745645-6rmsq" Jul 22 16:45:05.343781 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:05.343757 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745645-v8mq7" Jul 22 16:45:06.185385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:06.185358 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-059dda-29745645-6rmsq" Jul 22 16:45:06.185385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:06.185371 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-059dda-29745645-6rmsq" event={"ID":"7b9dec27-5e57-41a7-80f1-1e7ff8b70fc0","Type":"ContainerDied","Data":"af7f24a8dc5898298b4e224378efd6a3637c6257a2a3d660ab23b72673c87b43"} Jul 22 16:45:06.185607 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:06.185420 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af7f24a8dc5898298b4e224378efd6a3637c6257a2a3d660ab23b72673c87b43" Jul 22 16:45:06.186650 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:06.186630 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745645-v8mq7" Jul 22 16:45:06.186771 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:06.186664 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745645-v8mq7" event={"ID":"8ed74baf-4f05-465a-baab-ff90f7e2bde5","Type":"ContainerDied","Data":"e3cf6afd9e326ca2eeb2bad8d7912e8d2f24e872676640f14961248b85fb35d7"} Jul 22 16:45:06.186771 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:06.186727 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3cf6afd9e326ca2eeb2bad8d7912e8d2f24e872676640f14961248b85fb35d7" Jul 22 16:45:19.434043 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:45:19.434006 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:46:04.540613 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.540575 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm"] Jul 22 16:46:04.541052 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.540960 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b9dec27-5e57-41a7-80f1-1e7ff8b70fc0" containerName="cleanup" Jul 22 16:46:04.541052 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.540984 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b9dec27-5e57-41a7-80f1-1e7ff8b70fc0" containerName="cleanup" Jul 22 16:46:04.541052 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.541001 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed74baf-4f05-465a-baab-ff90f7e2bde5" containerName="cleanup" Jul 22 16:46:04.541052 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.541006 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed74baf-4f05-465a-baab-ff90f7e2bde5" containerName="cleanup" Jul 22 16:46:04.541200 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.541081 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ed74baf-4f05-465a-baab-ff90f7e2bde5" containerName="cleanup" Jul 22 16:46:04.541200 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.541089 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b9dec27-5e57-41a7-80f1-1e7ff8b70fc0" containerName="cleanup" Jul 22 16:46:04.543978 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.543957 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm" Jul 22 16:46:04.546548 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.546525 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfdj\"" Jul 22 16:46:04.549081 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.549059 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 16:46:04.561762 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.561738 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm"] Jul 22 16:46:04.713614 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.713581 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c97227-0b76-4c4c-9afb-b4cb42961eca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm\" (UID: \"52c97227-0b76-4c4c-9afb-b4cb42961eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm" Jul 22 16:46:04.815041 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.814960 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c97227-0b76-4c4c-9afb-b4cb42961eca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm\" (UID: \"52c97227-0b76-4c4c-9afb-b4cb42961eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm" Jul 22 16:46:04.815577 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.815556 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c97227-0b76-4c4c-9afb-b4cb42961eca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm\" (UID: \"52c97227-0b76-4c4c-9afb-b4cb42961eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm" Jul 22 16:46:04.854403 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.854368 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm" Jul 22 16:46:04.996809 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:04.996788 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm"] Jul 22 16:46:04.999481 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:46:04.999457 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c97227_0b76_4c4c_9afb_b4cb42961eca.slice/crio-664e6f27dd1efc3646b7c0335a2ac90023dba4e29b02387743bc198c44cbece9 WatchSource:0}: Error finding container 664e6f27dd1efc3646b7c0335a2ac90023dba4e29b02387743bc198c44cbece9: Status 404 returned error can't find the container with id 664e6f27dd1efc3646b7c0335a2ac90023dba4e29b02387743bc198c44cbece9 Jul 22 16:46:05.472004 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:05.471905 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm" event={"ID":"52c97227-0b76-4c4c-9afb-b4cb42961eca","Type":"ContainerStarted","Data":"92c3d02bb8375e1a89f5cdd12ebcac451ba75c1faf6384bfbd7a1c1a3cd0da7f"} Jul 22 16:46:05.472004 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:05.471967 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm" event={"ID":"52c97227-0b76-4c4c-9afb-b4cb42961eca","Type":"ContainerStarted","Data":"664e6f27dd1efc3646b7c0335a2ac90023dba4e29b02387743bc198c44cbece9"} Jul 22 16:46:05.492586 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:05.492529 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm" podStartSLOduration=1.4925131139999999 podStartE2EDuration="1.492513114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:46:05.490444124 +0000 UTC m=+2330.632318214" watchObservedRunningTime="2026-07-22 16:46:05.492513114 +0000 UTC m=+2330.634387192" Jul 22 16:46:06.478187 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:06.478153 2567 generic.go:356] "Generic (PLEG): container finished" podID="52c97227-0b76-4c4c-9afb-b4cb42961eca" containerID="92c3d02bb8375e1a89f5cdd12ebcac451ba75c1faf6384bfbd7a1c1a3cd0da7f" exitCode=0 Jul 22 16:46:06.478187 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:06.478192 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm" event={"ID":"52c97227-0b76-4c4c-9afb-b4cb42961eca","Type":"ContainerDied","Data":"92c3d02bb8375e1a89f5cdd12ebcac451ba75c1faf6384bfbd7a1c1a3cd0da7f"} Jul 22 16:46:07.619144 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:07.619119 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm" Jul 22 16:46:07.744585 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:07.744493 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c97227-0b76-4c4c-9afb-b4cb42961eca-maas-api-service-ca\") pod \"52c97227-0b76-4c4c-9afb-b4cb42961eca\" (UID: \"52c97227-0b76-4c4c-9afb-b4cb42961eca\") " Jul 22 16:46:07.744875 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:07.744851 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c97227-0b76-4c4c-9afb-b4cb42961eca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52c97227-0b76-4c4c-9afb-b4cb42961eca" (UID: "52c97227-0b76-4c4c-9afb-b4cb42961eca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:46:07.845444 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:07.845409 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c97227-0b76-4c4c-9afb-b4cb42961eca-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:46:08.490561 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:08.490534 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm" Jul 22 16:46:08.490766 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:08.490527 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm" event={"ID":"52c97227-0b76-4c4c-9afb-b4cb42961eca","Type":"ContainerDied","Data":"664e6f27dd1efc3646b7c0335a2ac90023dba4e29b02387743bc198c44cbece9"} Jul 22 16:46:08.490766 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:08.490660 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="664e6f27dd1efc3646b7c0335a2ac90023dba4e29b02387743bc198c44cbece9" Jul 22 16:46:08.597736 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:08.597700 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:46:19.702582 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:19.702550 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm"] Jul 22 16:46:19.708681 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:19.708648 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa170d-x6ktm"] Jul 22 16:46:19.983646 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:19.983559 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52c97227-0b76-4c4c-9afb-b4cb42961eca" path="/var/lib/kubelet/pods/52c97227-0b76-4c4c-9afb-b4cb42961eca/volumes" Jul 22 16:46:25.167902 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:25.167867 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:46:28.086578 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:28.086541 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9"] Jul 22 16:46:28.086985 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:28.086924 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c97227-0b76-4c4c-9afb-b4cb42961eca" containerName="revoke-keys" Jul 22 16:46:28.086985 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:28.086937 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c97227-0b76-4c4c-9afb-b4cb42961eca" containerName="revoke-keys" Jul 22 16:46:28.087058 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:28.087018 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c97227-0b76-4c4c-9afb-b4cb42961eca" containerName="revoke-keys" Jul 22 16:46:28.090119 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:28.090101 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9" Jul 22 16:46:28.092609 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:28.092586 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfdj\"" Jul 22 16:46:28.092983 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:28.092969 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 16:46:28.111745 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:28.111705 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9"] Jul 22 16:46:28.122676 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:28.122649 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422f40cf-914c-46ee-9f81-cee364d6ad62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9\" (UID: \"422f40cf-914c-46ee-9f81-cee364d6ad62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9" Jul 22 16:46:28.224184 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:28.224144 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422f40cf-914c-46ee-9f81-cee364d6ad62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9\" (UID: \"422f40cf-914c-46ee-9f81-cee364d6ad62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9" Jul 22 16:46:28.224779 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:28.224760 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422f40cf-914c-46ee-9f81-cee364d6ad62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9\" (UID: \"422f40cf-914c-46ee-9f81-cee364d6ad62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9" Jul 22 16:46:28.400605 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:28.400519 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9" Jul 22 16:46:28.556768 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:28.556743 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9"] Jul 22 16:46:28.559493 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:46:28.559463 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422f40cf_914c_46ee_9f81_cee364d6ad62.slice/crio-4dcd17a2f90565bd176aab8c17d7edeec197018e6ee176b5e71a6de1518de1ec WatchSource:0}: Error finding container 4dcd17a2f90565bd176aab8c17d7edeec197018e6ee176b5e71a6de1518de1ec: Status 404 returned error can't find the container with id 4dcd17a2f90565bd176aab8c17d7edeec197018e6ee176b5e71a6de1518de1ec Jul 22 16:46:28.591581 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:28.591550 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9" event={"ID":"422f40cf-914c-46ee-9f81-cee364d6ad62","Type":"ContainerStarted","Data":"4dcd17a2f90565bd176aab8c17d7edeec197018e6ee176b5e71a6de1518de1ec"} Jul 22 16:46:29.598388 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:29.598355 2567 generic.go:356] "Generic (PLEG): container finished" podID="422f40cf-914c-46ee-9f81-cee364d6ad62" containerID="54c1b9acc420d4aee80eaf1fcb049b3b25153607a2c457ed6b361d520a4c4f6c" exitCode=0 Jul 22 16:46:29.598840 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:29.598402 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9" event={"ID":"422f40cf-914c-46ee-9f81-cee364d6ad62","Type":"ContainerDied","Data":"54c1b9acc420d4aee80eaf1fcb049b3b25153607a2c457ed6b361d520a4c4f6c"} Jul 22 16:46:30.747754 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:30.747725 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9" Jul 22 16:46:30.847958 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:30.847928 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422f40cf-914c-46ee-9f81-cee364d6ad62-maas-api-service-ca\") pod \"422f40cf-914c-46ee-9f81-cee364d6ad62\" (UID: \"422f40cf-914c-46ee-9f81-cee364d6ad62\") " Jul 22 16:46:30.848297 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:30.848274 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422f40cf-914c-46ee-9f81-cee364d6ad62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "422f40cf-914c-46ee-9f81-cee364d6ad62" (UID: "422f40cf-914c-46ee-9f81-cee364d6ad62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:46:30.949141 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:30.949056 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422f40cf-914c-46ee-9f81-cee364d6ad62-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:46:31.608332 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:31.608301 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9" Jul 22 16:46:31.608516 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:31.608353 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9" event={"ID":"422f40cf-914c-46ee-9f81-cee364d6ad62","Type":"ContainerDied","Data":"4dcd17a2f90565bd176aab8c17d7edeec197018e6ee176b5e71a6de1518de1ec"} Jul 22 16:46:31.608516 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:31.608407 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dcd17a2f90565bd176aab8c17d7edeec197018e6ee176b5e71a6de1518de1ec" Jul 22 16:46:38.755601 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:38.755565 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-059dda-29745645-6rmsq"] Jul 22 16:46:38.784543 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:38.784510 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-059dda-29745645-6rmsq"] Jul 22 16:46:39.983678 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:39.983644 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b9dec27-5e57-41a7-80f1-1e7ff8b70fc0" path="/var/lib/kubelet/pods/7b9dec27-5e57-41a7-80f1-1e7ff8b70fc0/volumes" Jul 22 16:46:42.086771 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:42.086731 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:46:43.234844 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:43.234806 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9"] Jul 22 16:46:43.242420 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:43.242391 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-059dda-wzcq9"] Jul 22 16:46:43.983460 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:43.983418 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="422f40cf-914c-46ee-9f81-cee364d6ad62" path="/var/lib/kubelet/pods/422f40cf-914c-46ee-9f81-cee364d6ad62/volumes" Jul 22 16:46:51.370027 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:51.369990 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt"] Jul 22 16:46:51.370508 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:51.370196 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" podUID="e925ef1f-2c5f-4766-a9c7-cc0868a4f961" containerName="istio-proxy" containerID="cri-o://95e96fbd9ce2351b7844b56e746d45e51a1eea212e1af511dac2b9f700477ade" gracePeriod=30 Jul 22 16:46:56.625116 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.625089 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:46:56.736633 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.736544 2567 generic.go:356] "Generic (PLEG): container finished" podID="e925ef1f-2c5f-4766-a9c7-cc0868a4f961" containerID="95e96fbd9ce2351b7844b56e746d45e51a1eea212e1af511dac2b9f700477ade" exitCode=0 Jul 22 16:46:56.736633 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.736597 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" Jul 22 16:46:56.736896 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.736627 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" event={"ID":"e925ef1f-2c5f-4766-a9c7-cc0868a4f961","Type":"ContainerDied","Data":"95e96fbd9ce2351b7844b56e746d45e51a1eea212e1af511dac2b9f700477ade"} Jul 22 16:46:56.736896 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.736684 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt" event={"ID":"e925ef1f-2c5f-4766-a9c7-cc0868a4f961","Type":"ContainerDied","Data":"f9e6601634168677c27359a5b063944a088559d32058d51771400550a7c9feae"} Jul 22 16:46:56.736896 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.736727 2567 scope.go:117] "RemoveContainer" containerID="95e96fbd9ce2351b7844b56e746d45e51a1eea212e1af511dac2b9f700477ade" Jul 22 16:46:56.745364 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.745348 2567 scope.go:117] "RemoveContainer" containerID="3079734f71eae5bcf88a74e4e216fb74aa07132809455687c0c522926f4c6148" Jul 22 16:46:56.753584 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.753567 2567 scope.go:117] "RemoveContainer" containerID="95e96fbd9ce2351b7844b56e746d45e51a1eea212e1af511dac2b9f700477ade" Jul 22 16:46:56.753836 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:46:56.753818 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e96fbd9ce2351b7844b56e746d45e51a1eea212e1af511dac2b9f700477ade\": container with ID starting with 95e96fbd9ce2351b7844b56e746d45e51a1eea212e1af511dac2b9f700477ade not found: ID does not exist" containerID="95e96fbd9ce2351b7844b56e746d45e51a1eea212e1af511dac2b9f700477ade" Jul 22 16:46:56.753903 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.753845 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e96fbd9ce2351b7844b56e746d45e51a1eea212e1af511dac2b9f700477ade"} err="failed to get container status \"95e96fbd9ce2351b7844b56e746d45e51a1eea212e1af511dac2b9f700477ade\": rpc error: code = NotFound desc = could not find container \"95e96fbd9ce2351b7844b56e746d45e51a1eea212e1af511dac2b9f700477ade\": container with ID starting with 95e96fbd9ce2351b7844b56e746d45e51a1eea212e1af511dac2b9f700477ade not found: ID does not exist" Jul 22 16:46:56.753903 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.753862 2567 scope.go:117] "RemoveContainer" containerID="3079734f71eae5bcf88a74e4e216fb74aa07132809455687c0c522926f4c6148" Jul 22 16:46:56.754112 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:46:56.754094 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3079734f71eae5bcf88a74e4e216fb74aa07132809455687c0c522926f4c6148\": container with ID starting with 3079734f71eae5bcf88a74e4e216fb74aa07132809455687c0c522926f4c6148 not found: ID does not exist" containerID="3079734f71eae5bcf88a74e4e216fb74aa07132809455687c0c522926f4c6148" Jul 22 16:46:56.754172 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.754122 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3079734f71eae5bcf88a74e4e216fb74aa07132809455687c0c522926f4c6148"} err="failed to get container status \"3079734f71eae5bcf88a74e4e216fb74aa07132809455687c0c522926f4c6148\": rpc error: code = NotFound desc = could not find container \"3079734f71eae5bcf88a74e4e216fb74aa07132809455687c0c522926f4c6148\": container with ID starting with 3079734f71eae5bcf88a74e4e216fb74aa07132809455687c0c522926f4c6148 not found: ID does not exist" Jul 22 16:46:56.790469 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.790441 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-envoy\") pod \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " Jul 22 16:46:56.790591 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.790475 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-credential-socket\") pod \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " Jul 22 16:46:56.790591 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.790503 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istiod-ca-cert\") pod \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " Jul 22 16:46:56.790591 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.790545 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-data\") pod \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " Jul 22 16:46:56.790591 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.790573 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7p2s\" (UniqueName: \"kubernetes.io/projected/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-kube-api-access-m7p2s\") pod \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " Jul 22 16:46:56.790833 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.790619 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-podinfo\") pod \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " Jul 22 16:46:56.790833 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.790678 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-workload-certs\") pod \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " Jul 22 16:46:56.790833 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.790752 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-workload-socket\") pod \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " Jul 22 16:46:56.790833 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.790804 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-token\") pod \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\" (UID: \"e925ef1f-2c5f-4766-a9c7-cc0868a4f961\") " Jul 22 16:46:56.791040 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.790831 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e925ef1f-2c5f-4766-a9c7-cc0868a4f961" (UID: "e925ef1f-2c5f-4766-a9c7-cc0868a4f961"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:46:56.791040 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.790992 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e925ef1f-2c5f-4766-a9c7-cc0868a4f961" (UID: "e925ef1f-2c5f-4766-a9c7-cc0868a4f961"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:46:56.791153 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.791037 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e925ef1f-2c5f-4766-a9c7-cc0868a4f961" (UID: "e925ef1f-2c5f-4766-a9c7-cc0868a4f961"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:46:56.791153 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.791129 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:46:56.791153 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.791148 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:46:56.791317 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.791248 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:46:56.791317 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.791302 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e925ef1f-2c5f-4766-a9c7-cc0868a4f961" (UID: "e925ef1f-2c5f-4766-a9c7-cc0868a4f961"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:46:56.791497 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.791471 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-data" (OuterVolumeSpecName: "istio-data") pod "e925ef1f-2c5f-4766-a9c7-cc0868a4f961" (UID: "e925ef1f-2c5f-4766-a9c7-cc0868a4f961"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:46:56.792914 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.792887 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e925ef1f-2c5f-4766-a9c7-cc0868a4f961" (UID: "e925ef1f-2c5f-4766-a9c7-cc0868a4f961"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:46:56.793216 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.793197 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-token" (OuterVolumeSpecName: "istio-token") pod "e925ef1f-2c5f-4766-a9c7-cc0868a4f961" (UID: "e925ef1f-2c5f-4766-a9c7-cc0868a4f961"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:46:56.793293 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.793229 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-kube-api-access-m7p2s" (OuterVolumeSpecName: "kube-api-access-m7p2s") pod "e925ef1f-2c5f-4766-a9c7-cc0868a4f961" (UID: "e925ef1f-2c5f-4766-a9c7-cc0868a4f961"). InnerVolumeSpecName "kube-api-access-m7p2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:46:56.793353 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.793294 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e925ef1f-2c5f-4766-a9c7-cc0868a4f961" (UID: "e925ef1f-2c5f-4766-a9c7-cc0868a4f961"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 16:46:56.892030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.891993 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:46:56.892030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.892027 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7p2s\" (UniqueName: \"kubernetes.io/projected/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-kube-api-access-m7p2s\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:46:56.892030 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.892036 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:46:56.892278 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.892048 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:46:56.892278 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.892056 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:46:56.892278 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:56.892065 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e925ef1f-2c5f-4766-a9c7-cc0868a4f961-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:46:57.087402 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:57.087364 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt"] Jul 22 16:46:57.110428 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:57.110397 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-059dda-openshift-default-76c577c85c-ptclt"] Jul 22 16:46:57.984073 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:46:57.984042 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e925ef1f-2c5f-4766-a9c7-cc0868a4f961" path="/var/lib/kubelet/pods/e925ef1f-2c5f-4766-a9c7-cc0868a4f961/volumes" Jul 22 16:47:01.129728 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.129672 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:47:01.786278 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.786243 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b"] Jul 22 16:47:01.786584 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.786572 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e925ef1f-2c5f-4766-a9c7-cc0868a4f961" containerName="istio-proxy" Jul 22 16:47:01.786633 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.786586 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e925ef1f-2c5f-4766-a9c7-cc0868a4f961" containerName="istio-proxy" Jul 22 16:47:01.786633 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.786598 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e925ef1f-2c5f-4766-a9c7-cc0868a4f961" containerName="istio-proxy" Jul 22 16:47:01.786633 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.786603 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e925ef1f-2c5f-4766-a9c7-cc0868a4f961" containerName="istio-proxy" Jul 22 16:47:01.786633 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.786622 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="422f40cf-914c-46ee-9f81-cee364d6ad62" containerName="revoke-keys" Jul 22 16:47:01.786633 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.786627 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="422f40cf-914c-46ee-9f81-cee364d6ad62" containerName="revoke-keys" Jul 22 16:47:01.786816 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.786684 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="422f40cf-914c-46ee-9f81-cee364d6ad62" containerName="revoke-keys" Jul 22 16:47:01.786816 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.786709 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e925ef1f-2c5f-4766-a9c7-cc0868a4f961" containerName="istio-proxy" Jul 22 16:47:01.786816 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.786716 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e925ef1f-2c5f-4766-a9c7-cc0868a4f961" containerName="istio-proxy" Jul 22 16:47:01.791137 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.791116 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:01.797498 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:47:01.797463 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" is forbidden: User \"system:node:ip-10-0-131-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-131-21.ec2.internal' and this object" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:01.797629 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:47:01.797532 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-api-b-285a31-openshift-default-dockercfg-rvwpg\" is forbidden: User \"system:node:ip-10-0-131-21.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-131-21.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-api-b-285a31-openshift-default-dockercfg-rvwpg\"" type="*v1.Secret" Jul 22 16:47:01.865954 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.865920 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b"] Jul 22 16:47:01.940304 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.940256 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-workload-certs\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:01.940497 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.940311 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc54f668-1f7a-425c-a06c-9121004be35d-istio-podinfo\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:01.940497 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.940342 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-credential-socket\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:01.940497 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.940365 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj9xd\" (UniqueName: \"kubernetes.io/projected/cc54f668-1f7a-425c-a06c-9121004be35d-kube-api-access-zj9xd\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:01.940497 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.940457 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-workload-socket\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:01.940497 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.940478 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc54f668-1f7a-425c-a06c-9121004be35d-istiod-ca-cert\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:01.940771 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.940502 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-istio-data\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:01.940771 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.940564 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc54f668-1f7a-425c-a06c-9121004be35d-istio-token\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:01.940771 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:01.940607 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-istio-envoy\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.041894 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.041810 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc54f668-1f7a-425c-a06c-9121004be35d-istio-podinfo\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.041894 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.041846 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-credential-socket\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.041894 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.041865 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj9xd\" (UniqueName: \"kubernetes.io/projected/cc54f668-1f7a-425c-a06c-9121004be35d-kube-api-access-zj9xd\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.042181 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.041924 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-workload-socket\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.042181 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.041949 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc54f668-1f7a-425c-a06c-9121004be35d-istiod-ca-cert\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.042181 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.041975 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-istio-data\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.042181 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.042033 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc54f668-1f7a-425c-a06c-9121004be35d-istio-token\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.042181 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.042074 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-istio-envoy\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.042437 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.042229 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-workload-certs\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.042437 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.042305 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-credential-socket\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.042437 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.042367 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-workload-socket\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.042622 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.042598 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-istio-data\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.042759 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.042735 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc54f668-1f7a-425c-a06c-9121004be35d-istiod-ca-cert\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.042805 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.042685 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-workload-certs\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.044343 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.044324 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-istio-envoy\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.044430 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.044400 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc54f668-1f7a-425c-a06c-9121004be35d-istio-podinfo\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.063831 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.063805 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj9xd\" (UniqueName: \"kubernetes.io/projected/cc54f668-1f7a-425c-a06c-9121004be35d-kube-api-access-zj9xd\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:02.064542 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:02.064523 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc54f668-1f7a-425c-a06c-9121004be35d-istio-token\") pod \"e2e-api-b-285a31-openshift-default-5898964785-jrp6b\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:03.102441 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:03.102409 2567 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" secret="" err="failed to sync secret cache: timed out waiting for the condition" Jul 22 16:47:03.102901 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:03.102480 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:03.284008 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:03.283972 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b"] Jul 22 16:47:03.287619 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:47:03.287589 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc54f668_1f7a_425c_a06c_9121004be35d.slice/crio-ffd671f4c8cf4fe3c2183fffeb779992cf08b05eca2a6c0dd49143fee1112509 WatchSource:0}: Error finding container ffd671f4c8cf4fe3c2183fffeb779992cf08b05eca2a6c0dd49143fee1112509: Status 404 returned error can't find the container with id ffd671f4c8cf4fe3c2183fffeb779992cf08b05eca2a6c0dd49143fee1112509 Jul 22 16:47:03.290213 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:03.290171 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:47:03.290322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:03.290247 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:47:03.290322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:03.290278 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:47:03.343854 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:03.343833 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-285a31-openshift-default-dockercfg-rvwpg\"" Jul 22 16:47:03.774661 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:03.774620 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" event={"ID":"cc54f668-1f7a-425c-a06c-9121004be35d","Type":"ContainerStarted","Data":"ffefe526e2e5e8375912b0817c9436d86ebc1c2d64ab4d3a8126ff4a2b10ed99"} Jul 22 16:47:03.774854 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:03.774684 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" event={"ID":"cc54f668-1f7a-425c-a06c-9121004be35d","Type":"ContainerStarted","Data":"ffd671f4c8cf4fe3c2183fffeb779992cf08b05eca2a6c0dd49143fee1112509"} Jul 22 16:47:03.840510 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:03.840454 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" podStartSLOduration=2.840433916 podStartE2EDuration="2.840433916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:47:03.840073693 +0000 UTC m=+2388.981947771" watchObservedRunningTime="2026-07-22 16:47:03.840433916 +0000 UTC m=+2388.982307972" Jul 22 16:47:04.103578 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:04.103495 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:04.108167 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:04.108144 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:04.779322 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:04.779291 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:04.780361 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:04.780341 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:13.675873 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:13.675832 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:47:16.038171 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:16.038143 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:47:16.039699 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:16.039665 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:47:16.048094 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:16.048069 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:47:16.049610 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:16.049591 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:47:19.037749 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:19.037715 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4"] Jul 22 16:47:19.042389 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:19.042370 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4" Jul 22 16:47:19.049301 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:19.049276 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfdj\"" Jul 22 16:47:19.057306 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:47:19.057258 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4\" is forbidden: User \"system:node:ip-10-0-131-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-131-21.ec2.internal' and this object" podUID="fe2d40fa-dab7-47db-aadf-57d363b65e7e" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4" Jul 22 16:47:19.057435 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:47:19.057412 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-131-21.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-131-21.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 22 16:47:19.095141 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:19.095104 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2d40fa-dab7-47db-aadf-57d363b65e7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4\" (UID: \"fe2d40fa-dab7-47db-aadf-57d363b65e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4" Jul 22 16:47:19.124246 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:19.124149 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4"] Jul 22 16:47:19.196450 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:19.196412 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2d40fa-dab7-47db-aadf-57d363b65e7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4\" (UID: \"fe2d40fa-dab7-47db-aadf-57d363b65e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4" Jul 22 16:47:20.196637 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:47:20.196603 2567 configmap.go:193] Couldn't get configMap odh-ai-gateway-infra/openshift-service-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 22 16:47:20.197092 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:47:20.196755 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe2d40fa-dab7-47db-aadf-57d363b65e7e-maas-api-service-ca podName:fe2d40fa-dab7-47db-aadf-57d363b65e7e nodeName:}" failed. No retries permitted until 2026-07-22 16:47:20.696731129 +0000 UTC m=+2405.838605202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-service-ca" (UniqueName: "kubernetes.io/configmap/fe2d40fa-dab7-47db-aadf-57d363b65e7e-maas-api-service-ca") pod "maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4" (UID: "fe2d40fa-dab7-47db-aadf-57d363b65e7e") : failed to sync configmap cache: timed out waiting for the condition Jul 22 16:47:20.258241 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:20.258207 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 16:47:20.711655 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:20.711610 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2d40fa-dab7-47db-aadf-57d363b65e7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4\" (UID: \"fe2d40fa-dab7-47db-aadf-57d363b65e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4" Jul 22 16:47:20.712542 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:20.712518 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2d40fa-dab7-47db-aadf-57d363b65e7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4\" (UID: \"fe2d40fa-dab7-47db-aadf-57d363b65e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4" Jul 22 16:47:20.820947 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:20.820894 2567 patch_prober.go:28] interesting pod/e2e-api-b-285a31-openshift-default-5898964785-jrp6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 16:47:20.821121 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:20.820974 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 16:47:20.852938 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:20.852904 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4" Jul 22 16:47:21.017878 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:21.017822 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4"] Jul 22 16:47:21.021061 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:47:21.021033 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2d40fa_dab7_47db_aadf_57d363b65e7e.slice/crio-1a7ecedbfa0a451df109712b82d49ea6c4b28099ddad228aa05fb69c2f2f54bb WatchSource:0}: Error finding container 1a7ecedbfa0a451df109712b82d49ea6c4b28099ddad228aa05fb69c2f2f54bb: Status 404 returned error can't find the container with id 1a7ecedbfa0a451df109712b82d49ea6c4b28099ddad228aa05fb69c2f2f54bb Jul 22 16:47:21.868628 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:21.868596 2567 generic.go:356] "Generic (PLEG): container finished" podID="fe2d40fa-dab7-47db-aadf-57d363b65e7e" containerID="a7dbe1dc10c9c0d892a7f6974eb98eaf7064550e388027e1bbaf2f4cd93b2cbb" exitCode=0 Jul 22 16:47:21.869120 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:21.868684 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4" event={"ID":"fe2d40fa-dab7-47db-aadf-57d363b65e7e","Type":"ContainerDied","Data":"a7dbe1dc10c9c0d892a7f6974eb98eaf7064550e388027e1bbaf2f4cd93b2cbb"} Jul 22 16:47:21.869120 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:21.868766 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4" event={"ID":"fe2d40fa-dab7-47db-aadf-57d363b65e7e","Type":"ContainerStarted","Data":"1a7ecedbfa0a451df109712b82d49ea6c4b28099ddad228aa05fb69c2f2f54bb"} Jul 22 16:47:21.870395 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:21.870367 2567 generic.go:356] "Generic (PLEG): container finished" podID="cc54f668-1f7a-425c-a06c-9121004be35d" containerID="ffefe526e2e5e8375912b0817c9436d86ebc1c2d64ab4d3a8126ff4a2b10ed99" exitCode=0 Jul 22 16:47:21.870517 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:21.870425 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" event={"ID":"cc54f668-1f7a-425c-a06c-9121004be35d","Type":"ContainerDied","Data":"ffefe526e2e5e8375912b0817c9436d86ebc1c2d64ab4d3a8126ff4a2b10ed99"} Jul 22 16:47:21.870835 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:21.870817 2567 scope.go:117] "RemoveContainer" containerID="ffefe526e2e5e8375912b0817c9436d86ebc1c2d64ab4d3a8126ff4a2b10ed99" Jul 22 16:47:21.871983 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:21.871951 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:47:21.872054 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:21.872036 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:47:21.872095 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:21.872078 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:47:22.878436 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:22.878399 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" event={"ID":"cc54f668-1f7a-425c-a06c-9121004be35d","Type":"ContainerStarted","Data":"e557b94b1b30fe285a9b5a2df4567f1d85d10a8293eb47889c15bafd8b7e28cd"} Jul 22 16:47:23.035229 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:23.034880 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4" Jul 22 16:47:23.119107 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:23.119011 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:23.125715 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:23.119400 2567 patch_prober.go:28] interesting pod/e2e-api-b-285a31-openshift-default-5898964785-jrp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 22 16:47:23.125715 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:23.119450 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 22 16:47:23.134994 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:23.134911 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2d40fa-dab7-47db-aadf-57d363b65e7e-maas-api-service-ca\") pod \"fe2d40fa-dab7-47db-aadf-57d363b65e7e\" (UID: \"fe2d40fa-dab7-47db-aadf-57d363b65e7e\") " Jul 22 16:47:23.135299 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:23.135274 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe2d40fa-dab7-47db-aadf-57d363b65e7e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe2d40fa-dab7-47db-aadf-57d363b65e7e" (UID: "fe2d40fa-dab7-47db-aadf-57d363b65e7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:47:23.236369 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:23.236330 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2d40fa-dab7-47db-aadf-57d363b65e7e-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:47:23.885200 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:23.885163 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4" event={"ID":"fe2d40fa-dab7-47db-aadf-57d363b65e7e","Type":"ContainerDied","Data":"1a7ecedbfa0a451df109712b82d49ea6c4b28099ddad228aa05fb69c2f2f54bb"} Jul 22 16:47:23.885200 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:23.885188 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4" Jul 22 16:47:23.885610 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:23.885219 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a7ecedbfa0a451df109712b82d49ea6c4b28099ddad228aa05fb69c2f2f54bb" Jul 22 16:47:24.103731 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:24.103670 2567 patch_prober.go:28] interesting pod/e2e-api-b-285a31-openshift-default-5898964785-jrp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 22 16:47:24.103938 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:24.103769 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 22 16:47:25.103772 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:25.103729 2567 patch_prober.go:28] interesting pod/e2e-api-b-285a31-openshift-default-5898964785-jrp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 22 16:47:25.104278 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:25.103801 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 22 16:47:26.103181 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:26.103138 2567 patch_prober.go:28] interesting pod/e2e-api-b-285a31-openshift-default-5898964785-jrp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 22 16:47:26.103519 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:26.103485 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 22 16:47:27.103616 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:27.103569 2567 patch_prober.go:28] interesting pod/e2e-api-b-285a31-openshift-default-5898964785-jrp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 22 16:47:27.104128 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:27.103651 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 22 16:47:28.106667 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:28.106635 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:28.107065 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:28.107025 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:28.107676 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:28.107660 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:47:37.286674 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:37.286589 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:47:39.165932 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:39.165895 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4"] Jul 22 16:47:39.169889 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:39.169859 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0cd70-nzvq4"] Jul 22 16:47:39.983556 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:39.983519 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe2d40fa-dab7-47db-aadf-57d363b65e7e" path="/var/lib/kubelet/pods/fe2d40fa-dab7-47db-aadf-57d363b65e7e/volumes" Jul 22 16:47:48.157622 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:48.157585 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642"] Jul 22 16:47:48.158088 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:48.157943 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe2d40fa-dab7-47db-aadf-57d363b65e7e" containerName="revoke-keys" Jul 22 16:47:48.158088 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:48.157954 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe2d40fa-dab7-47db-aadf-57d363b65e7e" containerName="revoke-keys" Jul 22 16:47:48.158088 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:48.158011 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe2d40fa-dab7-47db-aadf-57d363b65e7e" containerName="revoke-keys" Jul 22 16:47:48.164162 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:48.164124 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642" Jul 22 16:47:48.169981 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:48.169948 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-285a31-5w642\" (UID: \"c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642" Jul 22 16:47:48.172246 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:48.172222 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfdj\"" Jul 22 16:47:48.172365 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:48.172331 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 16:47:48.190953 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:48.190918 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642"] Jul 22 16:47:48.271143 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:48.271103 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-285a31-5w642\" (UID: \"c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642" Jul 22 16:47:48.271825 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:48.271802 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-285a31-5w642\" (UID: \"c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642" Jul 22 16:47:48.474767 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:48.474650 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642" Jul 22 16:47:48.631765 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:48.631737 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642"] Jul 22 16:47:48.633865 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:47:48.633834 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6103c57_ff2e_429e_9c3f_0a9b8c19b1e1.slice/crio-67d2049fa10b3406e0312272fa4c4a32e4e054d20fb4a57c72b1879769836907 WatchSource:0}: Error finding container 67d2049fa10b3406e0312272fa4c4a32e4e054d20fb4a57c72b1879769836907: Status 404 returned error can't find the container with id 67d2049fa10b3406e0312272fa4c4a32e4e054d20fb4a57c72b1879769836907 Jul 22 16:47:49.007706 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:49.007661 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642" event={"ID":"c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1","Type":"ContainerStarted","Data":"1d2d1bc6978af204725ff8659a712d27868a925844b3dcda32aea86018b32a34"} Jul 22 16:47:49.007883 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:49.007751 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642" event={"ID":"c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1","Type":"ContainerStarted","Data":"67d2049fa10b3406e0312272fa4c4a32e4e054d20fb4a57c72b1879769836907"} Jul 22 16:47:49.031457 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:49.031391 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642" podStartSLOduration=1.031372662 podStartE2EDuration="1.031372662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:47:49.028965403 +0000 UTC m=+2434.170839495" watchObservedRunningTime="2026-07-22 16:47:49.031372662 +0000 UTC m=+2434.173246739" Jul 22 16:47:50.013952 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:50.013921 2567 generic.go:356] "Generic (PLEG): container finished" podID="c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1" containerID="1d2d1bc6978af204725ff8659a712d27868a925844b3dcda32aea86018b32a34" exitCode=0 Jul 22 16:47:50.014396 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:50.013971 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642" event={"ID":"c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1","Type":"ContainerDied","Data":"1d2d1bc6978af204725ff8659a712d27868a925844b3dcda32aea86018b32a34"} Jul 22 16:47:51.165481 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:51.165456 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642" Jul 22 16:47:51.198019 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:51.197982 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1-maas-api-service-ca\") pod \"c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1\" (UID: \"c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1\") " Jul 22 16:47:51.198349 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:51.198327 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1" (UID: "c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:47:51.299191 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:51.299098 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:47:52.024813 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:52.024781 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642" Jul 22 16:47:52.024813 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:52.024797 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642" event={"ID":"c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1","Type":"ContainerDied","Data":"67d2049fa10b3406e0312272fa4c4a32e4e054d20fb4a57c72b1879769836907"} Jul 22 16:47:52.025025 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:47:52.024850 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67d2049fa10b3406e0312272fa4c4a32e4e054d20fb4a57c72b1879769836907" Jul 22 16:48:08.136281 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:08.136242 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:48:08.224549 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:08.224509 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642"] Jul 22 16:48:08.227361 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:08.227332 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-285a31-5w642"] Jul 22 16:48:09.983745 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:09.983681 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1" path="/var/lib/kubelet/pods/c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1/volumes" Jul 22 16:48:14.748152 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:14.748118 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b"] Jul 22 16:48:14.748596 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:14.748378 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" containerName="istio-proxy" containerID="cri-o://e557b94b1b30fe285a9b5a2df4567f1d85d10a8293eb47889c15bafd8b7e28cd" gracePeriod=30 Jul 22 16:48:19.997884 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:19.997856 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:48:20.170448 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.170413 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-credential-socket\") pod \"cc54f668-1f7a-425c-a06c-9121004be35d\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " Jul 22 16:48:20.170654 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.170467 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj9xd\" (UniqueName: \"kubernetes.io/projected/cc54f668-1f7a-425c-a06c-9121004be35d-kube-api-access-zj9xd\") pod \"cc54f668-1f7a-425c-a06c-9121004be35d\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " Jul 22 16:48:20.170654 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.170507 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-workload-socket\") pod \"cc54f668-1f7a-425c-a06c-9121004be35d\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " Jul 22 16:48:20.170654 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.170545 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-workload-certs\") pod \"cc54f668-1f7a-425c-a06c-9121004be35d\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " Jul 22 16:48:20.170654 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.170587 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc54f668-1f7a-425c-a06c-9121004be35d-istio-token\") pod \"cc54f668-1f7a-425c-a06c-9121004be35d\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " Jul 22 16:48:20.170654 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.170615 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc54f668-1f7a-425c-a06c-9121004be35d-istiod-ca-cert\") pod \"cc54f668-1f7a-425c-a06c-9121004be35d\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " Jul 22 16:48:20.170956 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.170655 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc54f668-1f7a-425c-a06c-9121004be35d-istio-podinfo\") pod \"cc54f668-1f7a-425c-a06c-9121004be35d\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " Jul 22 16:48:20.170956 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.170679 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-istio-data\") pod \"cc54f668-1f7a-425c-a06c-9121004be35d\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " Jul 22 16:48:20.170956 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.170708 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cc54f668-1f7a-425c-a06c-9121004be35d" (UID: "cc54f668-1f7a-425c-a06c-9121004be35d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:48:20.170956 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.170736 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-istio-envoy\") pod \"cc54f668-1f7a-425c-a06c-9121004be35d\" (UID: \"cc54f668-1f7a-425c-a06c-9121004be35d\") " Jul 22 16:48:20.170956 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.170837 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cc54f668-1f7a-425c-a06c-9121004be35d" (UID: "cc54f668-1f7a-425c-a06c-9121004be35d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:48:20.170956 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.170937 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cc54f668-1f7a-425c-a06c-9121004be35d" (UID: "cc54f668-1f7a-425c-a06c-9121004be35d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:48:20.171397 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.171157 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:48:20.171397 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.171177 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:48:20.171397 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.171190 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:48:20.171397 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.171240 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc54f668-1f7a-425c-a06c-9121004be35d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cc54f668-1f7a-425c-a06c-9121004be35d" (UID: "cc54f668-1f7a-425c-a06c-9121004be35d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:48:20.172053 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.172021 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-istio-data" (OuterVolumeSpecName: "istio-data") pod "cc54f668-1f7a-425c-a06c-9121004be35d" (UID: "cc54f668-1f7a-425c-a06c-9121004be35d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:48:20.173060 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.173010 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc54f668-1f7a-425c-a06c-9121004be35d-kube-api-access-zj9xd" (OuterVolumeSpecName: "kube-api-access-zj9xd") pod "cc54f668-1f7a-425c-a06c-9121004be35d" (UID: "cc54f668-1f7a-425c-a06c-9121004be35d"). InnerVolumeSpecName "kube-api-access-zj9xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:48:20.173060 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.173026 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cc54f668-1f7a-425c-a06c-9121004be35d" (UID: "cc54f668-1f7a-425c-a06c-9121004be35d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:48:20.173259 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.173131 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc54f668-1f7a-425c-a06c-9121004be35d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cc54f668-1f7a-425c-a06c-9121004be35d" (UID: "cc54f668-1f7a-425c-a06c-9121004be35d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 16:48:20.173512 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.173481 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc54f668-1f7a-425c-a06c-9121004be35d-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc54f668-1f7a-425c-a06c-9121004be35d" (UID: "cc54f668-1f7a-425c-a06c-9121004be35d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:48:20.176186 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.176163 2567 generic.go:356] "Generic (PLEG): container finished" podID="cc54f668-1f7a-425c-a06c-9121004be35d" containerID="e557b94b1b30fe285a9b5a2df4567f1d85d10a8293eb47889c15bafd8b7e28cd" exitCode=0 Jul 22 16:48:20.176277 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.176217 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" Jul 22 16:48:20.176319 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.176210 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" event={"ID":"cc54f668-1f7a-425c-a06c-9121004be35d","Type":"ContainerDied","Data":"e557b94b1b30fe285a9b5a2df4567f1d85d10a8293eb47889c15bafd8b7e28cd"} Jul 22 16:48:20.176355 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.176334 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b" event={"ID":"cc54f668-1f7a-425c-a06c-9121004be35d","Type":"ContainerDied","Data":"ffd671f4c8cf4fe3c2183fffeb779992cf08b05eca2a6c0dd49143fee1112509"} Jul 22 16:48:20.176389 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.176361 2567 scope.go:117] "RemoveContainer" containerID="e557b94b1b30fe285a9b5a2df4567f1d85d10a8293eb47889c15bafd8b7e28cd" Jul 22 16:48:20.191725 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.191702 2567 scope.go:117] "RemoveContainer" containerID="ffefe526e2e5e8375912b0817c9436d86ebc1c2d64ab4d3a8126ff4a2b10ed99" Jul 22 16:48:20.200366 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.200209 2567 scope.go:117] "RemoveContainer" containerID="e557b94b1b30fe285a9b5a2df4567f1d85d10a8293eb47889c15bafd8b7e28cd" Jul 22 16:48:20.200474 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:48:20.200455 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e557b94b1b30fe285a9b5a2df4567f1d85d10a8293eb47889c15bafd8b7e28cd\": container with ID starting with e557b94b1b30fe285a9b5a2df4567f1d85d10a8293eb47889c15bafd8b7e28cd not found: ID does not exist" containerID="e557b94b1b30fe285a9b5a2df4567f1d85d10a8293eb47889c15bafd8b7e28cd" Jul 22 16:48:20.200525 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.200486 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e557b94b1b30fe285a9b5a2df4567f1d85d10a8293eb47889c15bafd8b7e28cd"} err="failed to get container status \"e557b94b1b30fe285a9b5a2df4567f1d85d10a8293eb47889c15bafd8b7e28cd\": rpc error: code = NotFound desc = could not find container \"e557b94b1b30fe285a9b5a2df4567f1d85d10a8293eb47889c15bafd8b7e28cd\": container with ID starting with e557b94b1b30fe285a9b5a2df4567f1d85d10a8293eb47889c15bafd8b7e28cd not found: ID does not exist" Jul 22 16:48:20.200525 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.200510 2567 scope.go:117] "RemoveContainer" containerID="ffefe526e2e5e8375912b0817c9436d86ebc1c2d64ab4d3a8126ff4a2b10ed99" Jul 22 16:48:20.200903 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:48:20.200839 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffefe526e2e5e8375912b0817c9436d86ebc1c2d64ab4d3a8126ff4a2b10ed99\": container with ID starting with ffefe526e2e5e8375912b0817c9436d86ebc1c2d64ab4d3a8126ff4a2b10ed99 not found: ID does not exist" containerID="ffefe526e2e5e8375912b0817c9436d86ebc1c2d64ab4d3a8126ff4a2b10ed99" Jul 22 16:48:20.200980 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.200912 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffefe526e2e5e8375912b0817c9436d86ebc1c2d64ab4d3a8126ff4a2b10ed99"} err="failed to get container status \"ffefe526e2e5e8375912b0817c9436d86ebc1c2d64ab4d3a8126ff4a2b10ed99\": rpc error: code = NotFound desc = could not find container \"ffefe526e2e5e8375912b0817c9436d86ebc1c2d64ab4d3a8126ff4a2b10ed99\": container with ID starting with ffefe526e2e5e8375912b0817c9436d86ebc1c2d64ab4d3a8126ff4a2b10ed99 not found: ID does not exist" Jul 22 16:48:20.206216 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.206070 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b"] Jul 22 16:48:20.213766 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.213736 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-285a31-openshift-default-5898964785-jrp6b"] Jul 22 16:48:20.271861 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.271822 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc54f668-1f7a-425c-a06c-9121004be35d-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:48:20.271861 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.271855 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc54f668-1f7a-425c-a06c-9121004be35d-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:48:20.271861 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.271865 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc54f668-1f7a-425c-a06c-9121004be35d-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:48:20.272174 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.271873 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:48:20.272174 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.271881 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc54f668-1f7a-425c-a06c-9121004be35d-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:48:20.272174 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:20.271891 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj9xd\" (UniqueName: \"kubernetes.io/projected/cc54f668-1f7a-425c-a06c-9121004be35d-kube-api-access-zj9xd\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:48:21.985017 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:21.984972 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" path="/var/lib/kubelet/pods/cc54f668-1f7a-425c-a06c-9121004be35d/volumes" Jul 22 16:48:24.233797 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.233759 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f"] Jul 22 16:48:24.234194 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.234134 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" containerName="istio-proxy" Jul 22 16:48:24.234194 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.234146 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" containerName="istio-proxy" Jul 22 16:48:24.234194 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.234155 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" containerName="istio-proxy" Jul 22 16:48:24.234194 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.234161 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" containerName="istio-proxy" Jul 22 16:48:24.234194 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.234181 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1" containerName="revoke-keys" Jul 22 16:48:24.234194 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.234187 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1" containerName="revoke-keys" Jul 22 16:48:24.234384 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.234255 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6103c57-ff2e-429e-9c3f-0a9b8c19b1e1" containerName="revoke-keys" Jul 22 16:48:24.234384 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.234265 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" containerName="istio-proxy" Jul 22 16:48:24.234384 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.234272 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc54f668-1f7a-425c-a06c-9121004be35d" containerName="istio-proxy" Jul 22 16:48:24.237449 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.237429 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.241212 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.241187 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-afe7a0-openshift-default-dockercfg-dr4fm\"" Jul 22 16:48:24.258993 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.258949 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f"] Jul 22 16:48:24.410845 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.410803 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-data\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.411047 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.410851 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfx44\" (UniqueName: \"kubernetes.io/projected/464bd6ee-320f-48d3-b167-5a2d356bcc10-kube-api-access-vfx44\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.411047 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.410927 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-workload-certs\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.411047 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.410950 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-podinfo\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.411047 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.411001 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-workload-socket\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.411208 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.411064 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-credential-socket\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.411208 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.411091 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-envoy\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.411208 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.411116 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464bd6ee-320f-48d3-b167-5a2d356bcc10-istiod-ca-cert\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.411208 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.411187 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-token\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.512648 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.512547 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-envoy\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.512648 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.512606 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464bd6ee-320f-48d3-b167-5a2d356bcc10-istiod-ca-cert\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.512648 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.512635 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-token\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.512983 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.512682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-data\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.512983 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.512741 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfx44\" (UniqueName: \"kubernetes.io/projected/464bd6ee-320f-48d3-b167-5a2d356bcc10-kube-api-access-vfx44\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.512983 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.512790 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-workload-certs\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.512983 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.512811 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-podinfo\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.512983 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.512836 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-workload-socket\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.512983 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.512889 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-credential-socket\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.513267 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.513155 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-data\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.513328 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.513296 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-credential-socket\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.513385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.513323 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-workload-certs\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.513444 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.513415 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-workload-socket\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.513712 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.513668 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464bd6ee-320f-48d3-b167-5a2d356bcc10-istiod-ca-cert\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.515138 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.515108 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-envoy\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.515602 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.515580 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-podinfo\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.523683 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.523651 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-token\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.523921 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.523899 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfx44\" (UniqueName: \"kubernetes.io/projected/464bd6ee-320f-48d3-b167-5a2d356bcc10-kube-api-access-vfx44\") pod \"e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.548577 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.548541 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:24.688325 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.688292 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f"] Jul 22 16:48:24.691425 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:48:24.691399 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464bd6ee_320f_48d3_b167_5a2d356bcc10.slice/crio-1c28decc2c7e766d62a391b10cea67efa25e3b25b134b92b763ede148933f010 WatchSource:0}: Error finding container 1c28decc2c7e766d62a391b10cea67efa25e3b25b134b92b763ede148933f010: Status 404 returned error can't find the container with id 1c28decc2c7e766d62a391b10cea67efa25e3b25b134b92b763ede148933f010 Jul 22 16:48:24.693535 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.693503 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:48:24.693599 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.693567 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:48:24.693644 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:24.693605 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:48:25.204774 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:25.204726 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" event={"ID":"464bd6ee-320f-48d3-b167-5a2d356bcc10","Type":"ContainerStarted","Data":"d13745c40d9edd83f2a40f4b9cf9b71e8f3d1eee241f871d3dd760b28e96c922"} Jul 22 16:48:25.204963 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:25.204803 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" event={"ID":"464bd6ee-320f-48d3-b167-5a2d356bcc10","Type":"ContainerStarted","Data":"1c28decc2c7e766d62a391b10cea67efa25e3b25b134b92b763ede148933f010"} Jul 22 16:48:25.230377 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:25.230321 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" podStartSLOduration=1.230305365 podStartE2EDuration="1.230305365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:48:25.228508896 +0000 UTC m=+2470.370382992" watchObservedRunningTime="2026-07-22 16:48:25.230305365 +0000 UTC m=+2470.372179443" Jul 22 16:48:25.548777 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:25.548731 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:25.553836 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:25.553812 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:26.209981 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:26.209945 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:26.211117 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:26.211096 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:26.539308 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:26.539270 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:48:37.242236 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:37.242196 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:48:41.211419 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:41.211383 2567 patch_prober.go:28] interesting pod/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 22 16:48:41.211896 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:41.211451 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" podUID="464bd6ee-320f-48d3-b167-5a2d356bcc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 22 16:48:41.288257 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:41.288225 2567 generic.go:356] "Generic (PLEG): container finished" podID="464bd6ee-320f-48d3-b167-5a2d356bcc10" containerID="d13745c40d9edd83f2a40f4b9cf9b71e8f3d1eee241f871d3dd760b28e96c922" exitCode=0 Jul 22 16:48:41.288438 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:41.288304 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" event={"ID":"464bd6ee-320f-48d3-b167-5a2d356bcc10","Type":"ContainerDied","Data":"d13745c40d9edd83f2a40f4b9cf9b71e8f3d1eee241f871d3dd760b28e96c922"} Jul 22 16:48:41.288736 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:41.288721 2567 scope.go:117] "RemoveContainer" containerID="d13745c40d9edd83f2a40f4b9cf9b71e8f3d1eee241f871d3dd760b28e96c922" Jul 22 16:48:41.289781 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:41.289744 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:48:41.289890 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:41.289808 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:48:41.289890 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:41.289837 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 16:48:42.295252 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:42.295207 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" event={"ID":"464bd6ee-320f-48d3-b167-5a2d356bcc10","Type":"ContainerStarted","Data":"a12b2392422532ea0771a76274bcec85f51d0e4ec3faa82371901bfd445d53ff"} Jul 22 16:48:42.549838 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:42.549745 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:42.550068 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:42.550044 2567 patch_prober.go:28] interesting pod/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 22 16:48:42.550151 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:42.550112 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" podUID="464bd6ee-320f-48d3-b167-5a2d356bcc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 22 16:48:43.549913 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:43.549872 2567 patch_prober.go:28] interesting pod/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 22 16:48:43.550290 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:43.549935 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" podUID="464bd6ee-320f-48d3-b167-5a2d356bcc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 22 16:48:44.549587 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:44.549547 2567 patch_prober.go:28] interesting pod/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 22 16:48:44.549773 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:44.549614 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" podUID="464bd6ee-320f-48d3-b167-5a2d356bcc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 22 16:48:45.136206 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:45.136158 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:48:45.549329 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:45.549287 2567 patch_prober.go:28] interesting pod/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 22 16:48:45.549513 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:45.549349 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" podUID="464bd6ee-320f-48d3-b167-5a2d356bcc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 22 16:48:46.549765 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:46.549726 2567 patch_prober.go:28] interesting pod/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 22 16:48:46.550171 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:46.549781 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" podUID="464bd6ee-320f-48d3-b167-5a2d356bcc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 22 16:48:47.554521 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:47.554487 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:47.555041 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:47.554912 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:48:47.555816 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:48:47.555793 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:49:34.356680 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:49:34.356638 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:49:53.832469 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:49:53.832436 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:50:11.740230 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:50:11.740194 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:50:27.431704 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:50:27.431660 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:50:44.437572 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:50:44.437535 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:50:58.655672 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:50:58.655631 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:51:13.443571 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:51:13.443534 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:51:16.157797 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:51:16.157769 2567 scope.go:117] "RemoveContainer" containerID="00e7ca1a1a1605c1bc19fe1c8967039c7cce8850cf7e5de1631d9c847d628afd" Jul 22 16:51:31.271585 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:51:31.271546 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:51:46.940239 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:51:46.940204 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:51:59.031894 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:51:59.031855 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:52:16.073302 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:16.073218 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:52:16.075841 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:16.075813 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:52:16.077385 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:16.077362 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:52:16.080103 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:16.080084 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g2pjb_33854df6-9972-4671-9749-2039970fd80c/ovn-acl-logging/0.log" Jul 22 16:52:16.177617 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:16.177583 2567 scope.go:117] "RemoveContainer" containerID="92c3d02bb8375e1a89f5cdd12ebcac451ba75c1faf6384bfbd7a1c1a3cd0da7f" Jul 22 16:52:19.545609 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.545575 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x"] Jul 22 16:52:19.550627 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.550606 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.553332 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.553312 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-afe7a0\"/\"openshift-service-ca.crt\"" Jul 22 16:52:19.554067 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.554046 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-afe7a0\"/\"kube-root-ca.crt\"" Jul 22 16:52:19.554354 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.554321 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-afe7a0\"/\"default-dockercfg-zlhgh\"" Jul 22 16:52:19.554459 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.554330 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-afe7a0\"/\"sub-test-model-afe7a0-kserve-self-signed-certs\"" Jul 22 16:52:19.557438 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.557414 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x"] Jul 22 16:52:19.646260 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.646219 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-tmp-dir\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.646450 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.646365 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-dshm\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.646450 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.646396 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-home\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.646450 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.646422 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-tls-certs\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.646554 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.646447 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-model-cache\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.646554 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.646481 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl879\" (UniqueName: \"kubernetes.io/projected/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-kube-api-access-wl879\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.747814 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.747778 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-dshm\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.747814 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.747813 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-home\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.748059 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.747839 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-tls-certs\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.748059 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.747863 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-model-cache\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.748059 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.747888 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl879\" (UniqueName: \"kubernetes.io/projected/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-kube-api-access-wl879\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.748059 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.747928 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-tmp-dir\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.748303 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.748282 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-model-cache\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.748381 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.748318 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-home\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.748381 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.748356 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-tmp-dir\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.750173 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.750150 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-dshm\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.750329 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.750313 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-tls-certs\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.756217 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.756199 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl879\" (UniqueName: \"kubernetes.io/projected/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-kube-api-access-wl879\") pod \"sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.860969 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.860873 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:19.997472 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:19.997427 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x"] Jul 22 16:52:20.001317 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:52:20.001289 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fe6dd16_7fd8_4aa5_a9e8_1339d014cc53.slice/crio-ec479808cdc1f3dab3cbc86ff6945480a00f2b9863691a265627f19098f07bef WatchSource:0}: Error finding container ec479808cdc1f3dab3cbc86ff6945480a00f2b9863691a265627f19098f07bef: Status 404 returned error can't find the container with id ec479808cdc1f3dab3cbc86ff6945480a00f2b9863691a265627f19098f07bef Jul 22 16:52:20.399909 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:20.399872 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" event={"ID":"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53","Type":"ContainerStarted","Data":"c70c4a3a2c17b7e319144d56079813cf0feb3cd932ab4fa2d04dfae596cd304b"} Jul 22 16:52:20.399909 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:20.399913 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" event={"ID":"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53","Type":"ContainerStarted","Data":"ec479808cdc1f3dab3cbc86ff6945480a00f2b9863691a265627f19098f07bef"} Jul 22 16:52:20.418035 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:20.417972 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" podStartSLOduration=1.417954023 podStartE2EDuration="1.417954023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:52:20.416828073 +0000 UTC m=+2705.558702130" watchObservedRunningTime="2026-07-22 16:52:20.417954023 +0000 UTC m=+2705.559828102" Jul 22 16:52:23.335291 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:23.335257 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:52:29.861734 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:29.861669 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:29.861734 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:29.861748 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:29.874465 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:29.874435 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:30.458848 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:30.458816 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:52:36.741290 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:36.741249 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:52:53.439380 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:53.439336 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:52:59.939002 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:52:59.938957 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x"] Jul 22 16:53:12.646451 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:12.646415 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:53:14.959483 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:14.959437 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" podUID="7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53" containerName="main" containerID="cri-o://c70c4a3a2c17b7e319144d56079813cf0feb3cd932ab4fa2d04dfae596cd304b" gracePeriod=45 Jul 22 16:53:15.212408 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.212344 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:53:15.248781 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.248745 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-model-cache\") pod \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " Jul 22 16:53:15.248963 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.248873 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-tls-certs\") pod \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " Jul 22 16:53:15.248963 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.248919 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-dshm\") pod \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " Jul 22 16:53:15.248963 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.248946 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-tmp-dir\") pod \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " Jul 22 16:53:15.249142 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.248978 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl879\" (UniqueName: \"kubernetes.io/projected/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-kube-api-access-wl879\") pod \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " Jul 22 16:53:15.249142 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.249011 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-home\") pod \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\" (UID: \"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53\") " Jul 22 16:53:15.249142 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.249018 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53" (UID: "7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:53:15.249305 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.249265 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53" (UID: "7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:53:15.249359 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.249314 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-model-cache\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:53:15.249359 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.249346 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-home" (OuterVolumeSpecName: "home") pod "7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53" (UID: "7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:53:15.251309 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.251280 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53" (UID: "7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:53:15.251309 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.251287 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-kube-api-access-wl879" (OuterVolumeSpecName: "kube-api-access-wl879") pod "7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53" (UID: "7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53"). InnerVolumeSpecName "kube-api-access-wl879". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:53:15.251451 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.251339 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-dshm" (OuterVolumeSpecName: "dshm") pod "7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53" (UID: "7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:53:15.350352 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.350315 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-tls-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:53:15.350352 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.350349 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-dshm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:53:15.350352 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.350362 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-tmp-dir\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:53:15.350598 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.350375 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl879\" (UniqueName: \"kubernetes.io/projected/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-kube-api-access-wl879\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:53:15.350598 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.350387 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53-home\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:53:15.675006 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.674969 2567 generic.go:356] "Generic (PLEG): container finished" podID="7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53" containerID="c70c4a3a2c17b7e319144d56079813cf0feb3cd932ab4fa2d04dfae596cd304b" exitCode=0 Jul 22 16:53:15.675181 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.675015 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" Jul 22 16:53:15.675181 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.675052 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" event={"ID":"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53","Type":"ContainerDied","Data":"c70c4a3a2c17b7e319144d56079813cf0feb3cd932ab4fa2d04dfae596cd304b"} Jul 22 16:53:15.675181 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.675099 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x" event={"ID":"7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53","Type":"ContainerDied","Data":"ec479808cdc1f3dab3cbc86ff6945480a00f2b9863691a265627f19098f07bef"} Jul 22 16:53:15.675181 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.675116 2567 scope.go:117] "RemoveContainer" containerID="c70c4a3a2c17b7e319144d56079813cf0feb3cd932ab4fa2d04dfae596cd304b" Jul 22 16:53:15.684233 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.684217 2567 scope.go:117] "RemoveContainer" containerID="c70c4a3a2c17b7e319144d56079813cf0feb3cd932ab4fa2d04dfae596cd304b" Jul 22 16:53:15.684470 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:53:15.684452 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70c4a3a2c17b7e319144d56079813cf0feb3cd932ab4fa2d04dfae596cd304b\": container with ID starting with c70c4a3a2c17b7e319144d56079813cf0feb3cd932ab4fa2d04dfae596cd304b not found: ID does not exist" containerID="c70c4a3a2c17b7e319144d56079813cf0feb3cd932ab4fa2d04dfae596cd304b" Jul 22 16:53:15.684508 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.684478 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70c4a3a2c17b7e319144d56079813cf0feb3cd932ab4fa2d04dfae596cd304b"} err="failed to get container status \"c70c4a3a2c17b7e319144d56079813cf0feb3cd932ab4fa2d04dfae596cd304b\": rpc error: code = NotFound desc = could not find container \"c70c4a3a2c17b7e319144d56079813cf0feb3cd932ab4fa2d04dfae596cd304b\": container with ID starting with c70c4a3a2c17b7e319144d56079813cf0feb3cd932ab4fa2d04dfae596cd304b not found: ID does not exist" Jul 22 16:53:15.696704 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.696670 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x"] Jul 22 16:53:15.700559 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.700537 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-afe7a0/sub-test-model-afe7a0-kserve-695ffbdbdc-2vh4x"] Jul 22 16:53:15.984272 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:15.984183 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53" path="/var/lib/kubelet/pods/7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53/volumes" Jul 22 16:53:16.198418 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:16.198384 2567 scope.go:117] "RemoveContainer" containerID="54c1b9acc420d4aee80eaf1fcb049b3b25153607a2c457ed6b361d520a4c4f6c" Jul 22 16:53:34.631739 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:34.631683 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:53:55.330793 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:53:55.330755 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:54:10.736306 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:10.736270 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:54:16.222899 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:16.222865 2567 scope.go:117] "RemoveContainer" containerID="1d2d1bc6978af204725ff8659a712d27868a925844b3dcda32aea86018b32a34" Jul 22 16:54:16.230912 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:16.230892 2567 scope.go:117] "RemoveContainer" containerID="a7dbe1dc10c9c0d892a7f6974eb98eaf7064550e388027e1bbaf2f4cd93b2cbb" Jul 22 16:54:25.623746 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:25.623709 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7694-140398118663200"] Jul 22 16:54:25.624137 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:25.624125 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53" containerName="main" Jul 22 16:54:25.624178 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:25.624138 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53" containerName="main" Jul 22 16:54:25.624231 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:25.624217 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe6dd16-7fd8-4aa5-a9e8-1339d014cc53" containerName="main" Jul 22 16:54:25.627212 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:25.627194 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-140398118663200" Jul 22 16:54:25.630482 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:25.630463 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tjvgd\"" Jul 22 16:54:25.633861 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:25.633836 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7694-140398118663200"] Jul 22 16:54:25.683776 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:25.683740 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppzrl\" (UniqueName: \"kubernetes.io/projected/af21f1af-7c98-4239-8d5f-382350ebf1f0-kube-api-access-ppzrl\") pod \"test-curl-7694-140398118663200\" (UID: \"af21f1af-7c98-4239-8d5f-382350ebf1f0\") " pod="opendatahub/test-curl-7694-140398118663200" Jul 22 16:54:25.784852 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:25.784803 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppzrl\" (UniqueName: \"kubernetes.io/projected/af21f1af-7c98-4239-8d5f-382350ebf1f0-kube-api-access-ppzrl\") pod \"test-curl-7694-140398118663200\" (UID: \"af21f1af-7c98-4239-8d5f-382350ebf1f0\") " pod="opendatahub/test-curl-7694-140398118663200" Jul 22 16:54:25.795383 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:25.795349 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppzrl\" (UniqueName: \"kubernetes.io/projected/af21f1af-7c98-4239-8d5f-382350ebf1f0-kube-api-access-ppzrl\") pod \"test-curl-7694-140398118663200\" (UID: \"af21f1af-7c98-4239-8d5f-382350ebf1f0\") " pod="opendatahub/test-curl-7694-140398118663200" Jul 22 16:54:25.938199 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:25.938109 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-140398118663200" Jul 22 16:54:26.075729 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:26.075673 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7694-140398118663200"] Jul 22 16:54:26.077535 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:54:26.077509 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf21f1af_7c98_4239_8d5f_382350ebf1f0.slice/crio-6d33944153658865da888196f0f88927c9c36b9a4caaeeea1de5ff76976dd7ec WatchSource:0}: Error finding container 6d33944153658865da888196f0f88927c9c36b9a4caaeeea1de5ff76976dd7ec: Status 404 returned error can't find the container with id 6d33944153658865da888196f0f88927c9c36b9a4caaeeea1de5ff76976dd7ec Jul 22 16:54:26.078744 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:26.078727 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:54:27.024240 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:27.024209 2567 generic.go:356] "Generic (PLEG): container finished" podID="af21f1af-7c98-4239-8d5f-382350ebf1f0" containerID="28bf4e07717f3e213f8b78b9170e0786fa6d944485606015ec9739cf4bcb8da0" exitCode=0 Jul 22 16:54:27.024611 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:27.024310 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7694-140398118663200" event={"ID":"af21f1af-7c98-4239-8d5f-382350ebf1f0","Type":"ContainerDied","Data":"28bf4e07717f3e213f8b78b9170e0786fa6d944485606015ec9739cf4bcb8da0"} Jul 22 16:54:27.024611 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:27.024351 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7694-140398118663200" event={"ID":"af21f1af-7c98-4239-8d5f-382350ebf1f0","Type":"ContainerStarted","Data":"6d33944153658865da888196f0f88927c9c36b9a4caaeeea1de5ff76976dd7ec"} Jul 22 16:54:28.169230 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.169206 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-140398118663200" Jul 22 16:54:28.213956 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.213928 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7694-140398118663200_af21f1af-7c98-4239-8d5f-382350ebf1f0/test-curl-7694-140398118663200/0.log" Jul 22 16:54:28.241373 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.241334 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7694-140398118663200"] Jul 22 16:54:28.245390 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.245361 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7694-140398118663200"] Jul 22 16:54:28.308827 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.308734 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppzrl\" (UniqueName: \"kubernetes.io/projected/af21f1af-7c98-4239-8d5f-382350ebf1f0-kube-api-access-ppzrl\") pod \"af21f1af-7c98-4239-8d5f-382350ebf1f0\" (UID: \"af21f1af-7c98-4239-8d5f-382350ebf1f0\") " Jul 22 16:54:28.310950 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.310928 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af21f1af-7c98-4239-8d5f-382350ebf1f0-kube-api-access-ppzrl" (OuterVolumeSpecName: "kube-api-access-ppzrl") pod "af21f1af-7c98-4239-8d5f-382350ebf1f0" (UID: "af21f1af-7c98-4239-8d5f-382350ebf1f0"). InnerVolumeSpecName "kube-api-access-ppzrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:54:28.375721 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.375672 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7694-140398118663200"] Jul 22 16:54:28.376047 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.376034 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af21f1af-7c98-4239-8d5f-382350ebf1f0" containerName="test-curl-7694-140398118663200" Jul 22 16:54:28.376089 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.376049 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="af21f1af-7c98-4239-8d5f-382350ebf1f0" containerName="test-curl-7694-140398118663200" Jul 22 16:54:28.376121 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.376111 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="af21f1af-7c98-4239-8d5f-382350ebf1f0" containerName="test-curl-7694-140398118663200" Jul 22 16:54:28.378905 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.378886 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-140398118663200" Jul 22 16:54:28.388869 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.388844 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7694-140398118663200"] Jul 22 16:54:28.409733 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.409679 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppzrl\" (UniqueName: \"kubernetes.io/projected/af21f1af-7c98-4239-8d5f-382350ebf1f0-kube-api-access-ppzrl\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:54:28.510155 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.510116 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhtwz\" (UniqueName: \"kubernetes.io/projected/e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10-kube-api-access-zhtwz\") pod \"test-curl-7694-140398118663200\" (UID: \"e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10\") " pod="opendatahub/test-curl-7694-140398118663200" Jul 22 16:54:28.611324 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.611234 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhtwz\" (UniqueName: \"kubernetes.io/projected/e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10-kube-api-access-zhtwz\") pod \"test-curl-7694-140398118663200\" (UID: \"e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10\") " pod="opendatahub/test-curl-7694-140398118663200" Jul 22 16:54:28.620398 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.620369 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhtwz\" (UniqueName: \"kubernetes.io/projected/e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10-kube-api-access-zhtwz\") pod \"test-curl-7694-140398118663200\" (UID: \"e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10\") " pod="opendatahub/test-curl-7694-140398118663200" Jul 22 16:54:28.688839 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.688796 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-140398118663200" Jul 22 16:54:28.826833 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:28.826799 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7694-140398118663200"] Jul 22 16:54:28.830004 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:54:28.829970 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9ed91b8_d6b8_47cf_8a2e_9cfd3d6e2c10.slice/crio-3b867c283c33e501b3041626bbc9e823d6ef7370dbc988d4d4ca51d757b4056a WatchSource:0}: Error finding container 3b867c283c33e501b3041626bbc9e823d6ef7370dbc988d4d4ca51d757b4056a: Status 404 returned error can't find the container with id 3b867c283c33e501b3041626bbc9e823d6ef7370dbc988d4d4ca51d757b4056a Jul 22 16:54:29.034538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:29.034507 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d33944153658865da888196f0f88927c9c36b9a4caaeeea1de5ff76976dd7ec" Jul 22 16:54:29.034538 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:29.034506 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-140398118663200" Jul 22 16:54:29.035683 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:29.035655 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7694-140398118663200" event={"ID":"e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10","Type":"ContainerStarted","Data":"3b867c283c33e501b3041626bbc9e823d6ef7370dbc988d4d4ca51d757b4056a"} Jul 22 16:54:29.047103 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:29.047063 2567 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7694-140398118663200" oldPodUID="af21f1af-7c98-4239-8d5f-382350ebf1f0" podUID="e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10" Jul 22 16:54:29.983614 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:29.983577 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af21f1af-7c98-4239-8d5f-382350ebf1f0" path="/var/lib/kubelet/pods/af21f1af-7c98-4239-8d5f-382350ebf1f0/volumes" Jul 22 16:54:30.041348 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:30.041315 2567 generic.go:356] "Generic (PLEG): container finished" podID="e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10" containerID="3924a6398150e31d598213e276e0bfb05a2464b257613c70f12efad9974ea7c4" exitCode=0 Jul 22 16:54:30.041515 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:30.041394 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7694-140398118663200" event={"ID":"e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10","Type":"ContainerDied","Data":"3924a6398150e31d598213e276e0bfb05a2464b257613c70f12efad9974ea7c4"} Jul 22 16:54:30.145126 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:30.145090 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:54:31.212489 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.212463 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-140398118663200" Jul 22 16:54:31.233368 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.233334 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7694-140398118663200_e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10/test-curl-7694-140398118663200/0.log" Jul 22 16:54:31.257283 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.257242 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7694-140398118663200"] Jul 22 16:54:31.262972 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.262944 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7694-140398118663200"] Jul 22 16:54:31.336885 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.336790 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhtwz\" (UniqueName: \"kubernetes.io/projected/e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10-kube-api-access-zhtwz\") pod \"e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10\" (UID: \"e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10\") " Jul 22 16:54:31.338884 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.338858 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10-kube-api-access-zhtwz" (OuterVolumeSpecName: "kube-api-access-zhtwz") pod "e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10" (UID: "e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10"). InnerVolumeSpecName "kube-api-access-zhtwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:54:31.388206 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.388169 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7694-140398118664208"] Jul 22 16:54:31.388521 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.388508 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10" containerName="test-curl-7694-140398118663200" Jul 22 16:54:31.388570 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.388522 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10" containerName="test-curl-7694-140398118663200" Jul 22 16:54:31.388608 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.388589 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10" containerName="test-curl-7694-140398118663200" Jul 22 16:54:31.391615 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.391599 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-140398118664208" Jul 22 16:54:31.395920 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.395893 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7694-140398118664208"] Jul 22 16:54:31.438386 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.438343 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhtwz\" (UniqueName: \"kubernetes.io/projected/e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10-kube-api-access-zhtwz\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:54:31.539783 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.539749 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7n8c\" (UniqueName: \"kubernetes.io/projected/19fea314-fe4e-47db-9003-9aa7f4bd94a0-kube-api-access-d7n8c\") pod \"test-curl-7694-140398118664208\" (UID: \"19fea314-fe4e-47db-9003-9aa7f4bd94a0\") " pod="opendatahub/test-curl-7694-140398118664208" Jul 22 16:54:31.640511 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.640419 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7n8c\" (UniqueName: \"kubernetes.io/projected/19fea314-fe4e-47db-9003-9aa7f4bd94a0-kube-api-access-d7n8c\") pod \"test-curl-7694-140398118664208\" (UID: \"19fea314-fe4e-47db-9003-9aa7f4bd94a0\") " pod="opendatahub/test-curl-7694-140398118664208" Jul 22 16:54:31.649182 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.649156 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7n8c\" (UniqueName: \"kubernetes.io/projected/19fea314-fe4e-47db-9003-9aa7f4bd94a0-kube-api-access-d7n8c\") pod \"test-curl-7694-140398118664208\" (UID: \"19fea314-fe4e-47db-9003-9aa7f4bd94a0\") " pod="opendatahub/test-curl-7694-140398118664208" Jul 22 16:54:31.701766 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.701732 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-140398118664208" Jul 22 16:54:31.835087 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.835059 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7694-140398118664208"] Jul 22 16:54:31.837264 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:54:31.837237 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19fea314_fe4e_47db_9003_9aa7f4bd94a0.slice/crio-7e576771fbd2d1d880b3ed407a57e6c65b6a152a2533acd2f1fb9da558c9bad5 WatchSource:0}: Error finding container 7e576771fbd2d1d880b3ed407a57e6c65b6a152a2533acd2f1fb9da558c9bad5: Status 404 returned error can't find the container with id 7e576771fbd2d1d880b3ed407a57e6c65b6a152a2533acd2f1fb9da558c9bad5 Jul 22 16:54:31.983948 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:31.983815 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10" path="/var/lib/kubelet/pods/e9ed91b8-d6b8-47cf-8a2e-9cfd3d6e2c10/volumes" Jul 22 16:54:32.052021 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:32.051989 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-140398118663200" Jul 22 16:54:32.052021 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:32.052000 2567 scope.go:117] "RemoveContainer" containerID="3924a6398150e31d598213e276e0bfb05a2464b257613c70f12efad9974ea7c4" Jul 22 16:54:32.053331 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:32.053276 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7694-140398118664208" event={"ID":"19fea314-fe4e-47db-9003-9aa7f4bd94a0","Type":"ContainerStarted","Data":"7e576771fbd2d1d880b3ed407a57e6c65b6a152a2533acd2f1fb9da558c9bad5"} Jul 22 16:54:33.059876 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:33.059840 2567 generic.go:356] "Generic (PLEG): container finished" podID="19fea314-fe4e-47db-9003-9aa7f4bd94a0" containerID="64a6ffbd830497ceef061a1e9b78754e6ce20cce227853f6a6ac5012a8ba42ad" exitCode=0 Jul 22 16:54:33.059876 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:33.059879 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7694-140398118664208" event={"ID":"19fea314-fe4e-47db-9003-9aa7f4bd94a0","Type":"ContainerDied","Data":"64a6ffbd830497ceef061a1e9b78754e6ce20cce227853f6a6ac5012a8ba42ad"} Jul 22 16:54:34.191811 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:34.191786 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-140398118664208" Jul 22 16:54:34.212048 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:34.212019 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7694-140398118664208_19fea314-fe4e-47db-9003-9aa7f4bd94a0/test-curl-7694-140398118664208/0.log" Jul 22 16:54:34.235731 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:34.235678 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7694-140398118664208"] Jul 22 16:54:34.237277 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:34.237254 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7694-140398118664208"] Jul 22 16:54:34.267508 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:34.267478 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7n8c\" (UniqueName: \"kubernetes.io/projected/19fea314-fe4e-47db-9003-9aa7f4bd94a0-kube-api-access-d7n8c\") pod \"19fea314-fe4e-47db-9003-9aa7f4bd94a0\" (UID: \"19fea314-fe4e-47db-9003-9aa7f4bd94a0\") " Jul 22 16:54:34.269583 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:34.269551 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fea314-fe4e-47db-9003-9aa7f4bd94a0-kube-api-access-d7n8c" (OuterVolumeSpecName: "kube-api-access-d7n8c") pod "19fea314-fe4e-47db-9003-9aa7f4bd94a0" (UID: "19fea314-fe4e-47db-9003-9aa7f4bd94a0"). InnerVolumeSpecName "kube-api-access-d7n8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:54:34.368959 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:34.368875 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7n8c\" (UniqueName: \"kubernetes.io/projected/19fea314-fe4e-47db-9003-9aa7f4bd94a0-kube-api-access-d7n8c\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:54:35.072871 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:35.072840 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-140398118664208" Jul 22 16:54:35.073073 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:35.072877 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e576771fbd2d1d880b3ed407a57e6c65b6a152a2533acd2f1fb9da558c9bad5" Jul 22 16:54:35.082726 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:54:35.082673 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7694-140398118664208\" is forbidden: User \"system:node:ip-10-0-131-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-21.ec2.internal' and this object" podUID="19fea314-fe4e-47db-9003-9aa7f4bd94a0" pod="opendatahub/test-curl-7694-140398118664208" Jul 22 16:54:35.981416 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:54:35.981376 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7694-140398118664208\" is forbidden: User \"system:node:ip-10-0-131-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-21.ec2.internal' and this object" podUID="19fea314-fe4e-47db-9003-9aa7f4bd94a0" pod="opendatahub/test-curl-7694-140398118664208" Jul 22 16:54:35.983375 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:35.983350 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19fea314-fe4e-47db-9003-9aa7f4bd94a0" path="/var/lib/kubelet/pods/19fea314-fe4e-47db-9003-9aa7f4bd94a0/volumes" Jul 22 16:54:37.297398 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:37.297360 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf"] Jul 22 16:54:37.297800 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:37.297743 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19fea314-fe4e-47db-9003-9aa7f4bd94a0" containerName="test-curl-7694-140398118664208" Jul 22 16:54:37.297800 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:37.297756 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="19fea314-fe4e-47db-9003-9aa7f4bd94a0" containerName="test-curl-7694-140398118664208" Jul 22 16:54:37.297884 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:37.297815 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="19fea314-fe4e-47db-9003-9aa7f4bd94a0" containerName="test-curl-7694-140398118664208" Jul 22 16:54:37.300826 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:37.300808 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf" Jul 22 16:54:37.303476 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:37.303454 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 16:54:37.303598 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:37.303489 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfdj\"" Jul 22 16:54:37.307655 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:37.307632 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf"] Jul 22 16:54:37.397682 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:37.397647 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d3a9aa-3f2c-4644-8aa9-219710cf15c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf\" (UID: \"96d3a9aa-3f2c-4644-8aa9-219710cf15c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf" Jul 22 16:54:37.498782 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:37.498745 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d3a9aa-3f2c-4644-8aa9-219710cf15c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf\" (UID: \"96d3a9aa-3f2c-4644-8aa9-219710cf15c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf" Jul 22 16:54:37.499384 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:37.499365 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d3a9aa-3f2c-4644-8aa9-219710cf15c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf\" (UID: \"96d3a9aa-3f2c-4644-8aa9-219710cf15c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf" Jul 22 16:54:37.612318 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:37.612216 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf" Jul 22 16:54:37.749979 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:37.749952 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf"] Jul 22 16:54:37.751799 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:54:37.751772 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d3a9aa_3f2c_4644_8aa9_219710cf15c1.slice/crio-6c06ea409e73a2b1c6f5a6d858756ebd80a019ace2a5f8c8fa60475d38793338 WatchSource:0}: Error finding container 6c06ea409e73a2b1c6f5a6d858756ebd80a019ace2a5f8c8fa60475d38793338: Status 404 returned error can't find the container with id 6c06ea409e73a2b1c6f5a6d858756ebd80a019ace2a5f8c8fa60475d38793338 Jul 22 16:54:38.089877 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:38.089836 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf" event={"ID":"96d3a9aa-3f2c-4644-8aa9-219710cf15c1","Type":"ContainerStarted","Data":"131862b38c95e7572441b26301d17669fcc8e588b9197fdf41e9fcd99e1db50a"} Jul 22 16:54:38.090066 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:38.089905 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf" event={"ID":"96d3a9aa-3f2c-4644-8aa9-219710cf15c1","Type":"ContainerStarted","Data":"6c06ea409e73a2b1c6f5a6d858756ebd80a019ace2a5f8c8fa60475d38793338"} Jul 22 16:54:38.109034 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:38.108982 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf" podStartSLOduration=1.108963053 podStartE2EDuration="1.108963053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:54:38.106673689 +0000 UTC m=+2843.248547770" watchObservedRunningTime="2026-07-22 16:54:38.108963053 +0000 UTC m=+2843.250837130" Jul 22 16:54:39.096589 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:39.096550 2567 generic.go:356] "Generic (PLEG): container finished" podID="96d3a9aa-3f2c-4644-8aa9-219710cf15c1" containerID="131862b38c95e7572441b26301d17669fcc8e588b9197fdf41e9fcd99e1db50a" exitCode=0 Jul 22 16:54:39.097019 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:39.096637 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf" event={"ID":"96d3a9aa-3f2c-4644-8aa9-219710cf15c1","Type":"ContainerDied","Data":"131862b38c95e7572441b26301d17669fcc8e588b9197fdf41e9fcd99e1db50a"} Jul 22 16:54:40.237120 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:40.237088 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf" Jul 22 16:54:40.326528 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:40.326487 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d3a9aa-3f2c-4644-8aa9-219710cf15c1-maas-api-service-ca\") pod \"96d3a9aa-3f2c-4644-8aa9-219710cf15c1\" (UID: \"96d3a9aa-3f2c-4644-8aa9-219710cf15c1\") " Jul 22 16:54:40.327031 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:40.327001 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d3a9aa-3f2c-4644-8aa9-219710cf15c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96d3a9aa-3f2c-4644-8aa9-219710cf15c1" (UID: "96d3a9aa-3f2c-4644-8aa9-219710cf15c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:54:40.427925 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:40.427838 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d3a9aa-3f2c-4644-8aa9-219710cf15c1-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:54:41.107251 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:41.107216 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf" Jul 22 16:54:41.107251 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:41.107237 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf" event={"ID":"96d3a9aa-3f2c-4644-8aa9-219710cf15c1","Type":"ContainerDied","Data":"6c06ea409e73a2b1c6f5a6d858756ebd80a019ace2a5f8c8fa60475d38793338"} Jul 22 16:54:41.107486 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:41.107289 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c06ea409e73a2b1c6f5a6d858756ebd80a019ace2a5f8c8fa60475d38793338" Jul 22 16:54:50.929515 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:50.929477 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:54:52.385790 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:52.385750 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf"] Jul 22 16:54:52.389570 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:52.389542 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff2a05-k4gmf"] Jul 22 16:54:53.983549 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:53.983512 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96d3a9aa-3f2c-4644-8aa9-219710cf15c1" path="/var/lib/kubelet/pods/96d3a9aa-3f2c-4644-8aa9-219710cf15c1/volumes" Jul 22 16:54:59.664413 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:59.664371 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78"] Jul 22 16:54:59.664961 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:59.664942 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d3a9aa-3f2c-4644-8aa9-219710cf15c1" containerName="revoke-keys" Jul 22 16:54:59.665043 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:59.664964 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d3a9aa-3f2c-4644-8aa9-219710cf15c1" containerName="revoke-keys" Jul 22 16:54:59.665096 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:59.665061 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d3a9aa-3f2c-4644-8aa9-219710cf15c1" containerName="revoke-keys" Jul 22 16:54:59.670216 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:59.670194 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78" Jul 22 16:54:59.672704 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:59.672669 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfdj\"" Jul 22 16:54:59.672853 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:59.672818 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 16:54:59.676283 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:59.676255 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78"] Jul 22 16:54:59.702092 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:59.702057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78\" (UID: \"3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78" Jul 22 16:54:59.802866 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:59.802827 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78\" (UID: \"3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78" Jul 22 16:54:59.803507 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:59.803486 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78\" (UID: \"3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78" Jul 22 16:54:59.982603 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:54:59.982521 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78" Jul 22 16:55:00.120001 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:00.119975 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78"] Jul 22 16:55:00.122448 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:55:00.122422 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a01d83a_b1ae_4f94_bedf_fd7e24cb2b13.slice/crio-c344658d8a17bca55357125f26d9ba476a5bcb0a7dd137f135ac47327f8ced3d WatchSource:0}: Error finding container c344658d8a17bca55357125f26d9ba476a5bcb0a7dd137f135ac47327f8ced3d: Status 404 returned error can't find the container with id c344658d8a17bca55357125f26d9ba476a5bcb0a7dd137f135ac47327f8ced3d Jul 22 16:55:00.196634 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:00.196600 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78" event={"ID":"3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13","Type":"ContainerStarted","Data":"c344658d8a17bca55357125f26d9ba476a5bcb0a7dd137f135ac47327f8ced3d"} Jul 22 16:55:01.203359 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:01.203314 2567 generic.go:356] "Generic (PLEG): container finished" podID="3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13" containerID="898cae9eb671afb4e9fe498491aff9e3a05d1d42885b1f9dec8536f56452682b" exitCode=0 Jul 22 16:55:01.203795 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:01.203359 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78" event={"ID":"3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13","Type":"ContainerDied","Data":"898cae9eb671afb4e9fe498491aff9e3a05d1d42885b1f9dec8536f56452682b"} Jul 22 16:55:02.341383 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:02.341359 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78" Jul 22 16:55:02.428618 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:02.428585 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13-maas-api-service-ca\") pod \"3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13\" (UID: \"3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13\") " Jul 22 16:55:02.428993 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:02.428969 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13" (UID: "3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:55:02.529340 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:02.529305 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:55:03.213810 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:03.213768 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78" event={"ID":"3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13","Type":"ContainerDied","Data":"c344658d8a17bca55357125f26d9ba476a5bcb0a7dd137f135ac47327f8ced3d"} Jul 22 16:55:03.213810 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:03.213793 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78" Jul 22 16:55:03.214018 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:03.213828 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c344658d8a17bca55357125f26d9ba476a5bcb0a7dd137f135ac47327f8ced3d" Jul 22 16:55:12.843719 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:12.843617 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nhksq"] Jul 22 16:55:14.760430 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:14.760391 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78"] Jul 22 16:55:14.764721 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:14.764679 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afe7a0-rlx78"] Jul 22 16:55:15.984225 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:15.984187 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13" path="/var/lib/kubelet/pods/3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13/volumes" Jul 22 16:55:21.389257 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:21.389222 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f"] Jul 22 16:55:21.389761 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:21.389455 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" podUID="464bd6ee-320f-48d3-b167-5a2d356bcc10" containerName="istio-proxy" containerID="cri-o://a12b2392422532ea0771a76274bcec85f51d0e4ec3faa82371901bfd445d53ff" gracePeriod=30 Jul 22 16:55:26.655161 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.655098 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:55:26.760027 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.759993 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-envoy\") pod \"464bd6ee-320f-48d3-b167-5a2d356bcc10\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " Jul 22 16:55:26.760218 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.760051 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-token\") pod \"464bd6ee-320f-48d3-b167-5a2d356bcc10\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " Jul 22 16:55:26.760218 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.760076 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfx44\" (UniqueName: \"kubernetes.io/projected/464bd6ee-320f-48d3-b167-5a2d356bcc10-kube-api-access-vfx44\") pod \"464bd6ee-320f-48d3-b167-5a2d356bcc10\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " Jul 22 16:55:26.760218 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.760093 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-podinfo\") pod \"464bd6ee-320f-48d3-b167-5a2d356bcc10\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " Jul 22 16:55:26.760218 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.760117 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-credential-socket\") pod \"464bd6ee-320f-48d3-b167-5a2d356bcc10\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " Jul 22 16:55:26.760218 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.760136 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464bd6ee-320f-48d3-b167-5a2d356bcc10-istiod-ca-cert\") pod \"464bd6ee-320f-48d3-b167-5a2d356bcc10\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " Jul 22 16:55:26.760218 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.760153 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-data\") pod \"464bd6ee-320f-48d3-b167-5a2d356bcc10\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " Jul 22 16:55:26.760532 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.760221 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-workload-certs\") pod \"464bd6ee-320f-48d3-b167-5a2d356bcc10\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " Jul 22 16:55:26.760532 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.760258 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-workload-socket\") pod \"464bd6ee-320f-48d3-b167-5a2d356bcc10\" (UID: \"464bd6ee-320f-48d3-b167-5a2d356bcc10\") " Jul 22 16:55:26.760532 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.760493 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "464bd6ee-320f-48d3-b167-5a2d356bcc10" (UID: "464bd6ee-320f-48d3-b167-5a2d356bcc10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:55:26.760714 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.760669 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "464bd6ee-320f-48d3-b167-5a2d356bcc10" (UID: "464bd6ee-320f-48d3-b167-5a2d356bcc10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:55:26.760784 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.760591 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464bd6ee-320f-48d3-b167-5a2d356bcc10-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "464bd6ee-320f-48d3-b167-5a2d356bcc10" (UID: "464bd6ee-320f-48d3-b167-5a2d356bcc10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:55:26.760851 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.760794 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "464bd6ee-320f-48d3-b167-5a2d356bcc10" (UID: "464bd6ee-320f-48d3-b167-5a2d356bcc10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:55:26.761280 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.761258 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-data" (OuterVolumeSpecName: "istio-data") pod "464bd6ee-320f-48d3-b167-5a2d356bcc10" (UID: "464bd6ee-320f-48d3-b167-5a2d356bcc10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:55:26.762498 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.762465 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-token" (OuterVolumeSpecName: "istio-token") pod "464bd6ee-320f-48d3-b167-5a2d356bcc10" (UID: "464bd6ee-320f-48d3-b167-5a2d356bcc10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:55:26.762678 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.762661 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "464bd6ee-320f-48d3-b167-5a2d356bcc10" (UID: "464bd6ee-320f-48d3-b167-5a2d356bcc10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 16:55:26.762816 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.762798 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "464bd6ee-320f-48d3-b167-5a2d356bcc10" (UID: "464bd6ee-320f-48d3-b167-5a2d356bcc10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:55:26.762880 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.762814 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464bd6ee-320f-48d3-b167-5a2d356bcc10-kube-api-access-vfx44" (OuterVolumeSpecName: "kube-api-access-vfx44") pod "464bd6ee-320f-48d3-b167-5a2d356bcc10" (UID: "464bd6ee-320f-48d3-b167-5a2d356bcc10"). InnerVolumeSpecName "kube-api-access-vfx44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:55:26.861959 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.861927 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:55:26.861959 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.861955 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:55:26.861959 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.861964 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfx44\" (UniqueName: \"kubernetes.io/projected/464bd6ee-320f-48d3-b167-5a2d356bcc10-kube-api-access-vfx44\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:55:26.861959 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.861974 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:55:26.862236 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.861984 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:55:26.862236 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.861992 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464bd6ee-320f-48d3-b167-5a2d356bcc10-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:55:26.862236 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.862002 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:55:26.862236 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.862009 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:55:26.862236 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:26.862017 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464bd6ee-320f-48d3-b167-5a2d356bcc10-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Jul 22 16:55:27.334097 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:27.334064 2567 generic.go:356] "Generic (PLEG): container finished" podID="464bd6ee-320f-48d3-b167-5a2d356bcc10" containerID="a12b2392422532ea0771a76274bcec85f51d0e4ec3faa82371901bfd445d53ff" exitCode=0 Jul 22 16:55:27.334275 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:27.334111 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" Jul 22 16:55:27.334275 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:27.334137 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" event={"ID":"464bd6ee-320f-48d3-b167-5a2d356bcc10","Type":"ContainerDied","Data":"a12b2392422532ea0771a76274bcec85f51d0e4ec3faa82371901bfd445d53ff"} Jul 22 16:55:27.334275 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:27.334186 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f" event={"ID":"464bd6ee-320f-48d3-b167-5a2d356bcc10","Type":"ContainerDied","Data":"1c28decc2c7e766d62a391b10cea67efa25e3b25b134b92b763ede148933f010"} Jul 22 16:55:27.334275 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:27.334210 2567 scope.go:117] "RemoveContainer" containerID="a12b2392422532ea0771a76274bcec85f51d0e4ec3faa82371901bfd445d53ff" Jul 22 16:55:27.343610 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:27.343589 2567 scope.go:117] "RemoveContainer" containerID="d13745c40d9edd83f2a40f4b9cf9b71e8f3d1eee241f871d3dd760b28e96c922" Jul 22 16:55:27.351743 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:27.351721 2567 scope.go:117] "RemoveContainer" containerID="a12b2392422532ea0771a76274bcec85f51d0e4ec3faa82371901bfd445d53ff" Jul 22 16:55:27.351989 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:55:27.351971 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12b2392422532ea0771a76274bcec85f51d0e4ec3faa82371901bfd445d53ff\": container with ID starting with a12b2392422532ea0771a76274bcec85f51d0e4ec3faa82371901bfd445d53ff not found: ID does not exist" containerID="a12b2392422532ea0771a76274bcec85f51d0e4ec3faa82371901bfd445d53ff" Jul 22 16:55:27.352042 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:27.351998 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12b2392422532ea0771a76274bcec85f51d0e4ec3faa82371901bfd445d53ff"} err="failed to get container status \"a12b2392422532ea0771a76274bcec85f51d0e4ec3faa82371901bfd445d53ff\": rpc error: code = NotFound desc = could not find container \"a12b2392422532ea0771a76274bcec85f51d0e4ec3faa82371901bfd445d53ff\": container with ID starting with a12b2392422532ea0771a76274bcec85f51d0e4ec3faa82371901bfd445d53ff not found: ID does not exist" Jul 22 16:55:27.352042 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:27.352016 2567 scope.go:117] "RemoveContainer" containerID="d13745c40d9edd83f2a40f4b9cf9b71e8f3d1eee241f871d3dd760b28e96c922" Jul 22 16:55:27.352266 ip-10-0-131-21 kubenswrapper[2567]: E0722 16:55:27.352249 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13745c40d9edd83f2a40f4b9cf9b71e8f3d1eee241f871d3dd760b28e96c922\": container with ID starting with d13745c40d9edd83f2a40f4b9cf9b71e8f3d1eee241f871d3dd760b28e96c922 not found: ID does not exist" containerID="d13745c40d9edd83f2a40f4b9cf9b71e8f3d1eee241f871d3dd760b28e96c922" Jul 22 16:55:27.352328 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:27.352278 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13745c40d9edd83f2a40f4b9cf9b71e8f3d1eee241f871d3dd760b28e96c922"} err="failed to get container status \"d13745c40d9edd83f2a40f4b9cf9b71e8f3d1eee241f871d3dd760b28e96c922\": rpc error: code = NotFound desc = could not find container \"d13745c40d9edd83f2a40f4b9cf9b71e8f3d1eee241f871d3dd760b28e96c922\": container with ID starting with d13745c40d9edd83f2a40f4b9cf9b71e8f3d1eee241f871d3dd760b28e96c922 not found: ID does not exist" Jul 22 16:55:27.356061 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:27.356037 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f"] Jul 22 16:55:27.361972 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:27.361950 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-afe7a0-openshift-default-f8d55445f-kkg4f"] Jul 22 16:55:27.983507 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:27.983469 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="464bd6ee-320f-48d3-b167-5a2d356bcc10" path="/var/lib/kubelet/pods/464bd6ee-320f-48d3-b167-5a2d356bcc10/volumes" Jul 22 16:55:28.280847 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:28.280810 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-587bf8b54-qg9rw_239e7a15-399f-4a8a-8341-edb28cf3faa4/manager/0.log" Jul 22 16:55:28.288281 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:28.288260 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-587bf8b54-qg9rw_239e7a15-399f-4a8a-8341-edb28cf3faa4/copy-manifests/0.log" Jul 22 16:55:28.876503 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:28.876476 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745645-v8mq7_8ed74baf-4f05-465a-baab-ff90f7e2bde5/cleanup/0.log" Jul 22 16:55:31.051191 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:31.051163 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nhksq_704997ce-7891-4120-a9ca-a114f41ed771/limitador/0.log" Jul 22 16:55:31.525820 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:31.525794 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678ndvrh_85f1c726-74bd-42a4-8f0d-342ac1e83aad/istio-proxy/0.log" Jul 22 16:55:31.652376 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:31.652348 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-8d5b7_252f357d-2451-4e46-9887-075ccae6dd38/discovery/0.log" Jul 22 16:55:32.401388 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:32.401359 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-586bddb798-57m2m_57dbd4ff-f4ed-484a-801b-9cef8a961409/router/0.log" Jul 22 16:55:33.235112 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:33.235079 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-78dsp_28240751-c131-445a-b78a-4ecf4dbcf4e3/main/0.log" Jul 22 16:55:41.631300 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:41.631269 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f7rz8_a19af359-cd18-4594-817d-a115665176bd/global-pull-secret-syncer/0.log" Jul 22 16:55:41.726186 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:41.726149 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vbpf9_278b6285-2b8e-4fc3-a861-2f6088d38721/konnectivity-agent/0.log" Jul 22 16:55:41.781035 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:41.781003 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-21.ec2.internal_f338029658e35a32ff779b9f1d620fb0/haproxy/0.log" Jul 22 16:55:47.402003 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:47.401970 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nhksq_704997ce-7891-4120-a9ca-a114f41ed771/limitador/0.log" Jul 22 16:55:49.488089 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:49.488056 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-rg4vq_fa7a9636-f842-48f7-ae45-dfcca3c01ee3/cluster-monitoring-operator/0.log" Jul 22 16:55:49.525274 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:49.525241 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-82pl8_e956f24d-9f22-4042-977e-e455a073722a/kube-state-metrics/0.log" Jul 22 16:55:49.545442 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:49.545412 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-82pl8_e956f24d-9f22-4042-977e-e455a073722a/kube-rbac-proxy-main/0.log" Jul 22 16:55:49.565958 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:49.565927 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-82pl8_e956f24d-9f22-4042-977e-e455a073722a/kube-rbac-proxy-self/0.log" Jul 22 16:55:49.593206 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:49.593179 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6b8df8fc77-n9zx4_2f07ce17-c736-4e6f-bd59-f6c84de3b161/metrics-server/0.log" Jul 22 16:55:49.653171 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:49.653140 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8p79m_dce4f379-8057-4575-afa2-08b971bf9c79/node-exporter/0.log" Jul 22 16:55:49.675525 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:49.675503 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8p79m_dce4f379-8057-4575-afa2-08b971bf9c79/kube-rbac-proxy/0.log" Jul 22 16:55:49.700475 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:49.700453 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8p79m_dce4f379-8057-4575-afa2-08b971bf9c79/init-textfile/0.log" Jul 22 16:55:50.227413 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:50.227377 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-h26ps_178d03dd-d01a-40a2-a753-9cb23e70d8d1/prometheus-operator/0.log" Jul 22 16:55:50.255164 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:50.255140 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-h26ps_178d03dd-d01a-40a2-a753-9cb23e70d8d1/kube-rbac-proxy/0.log" Jul 22 16:55:50.311038 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:50.311008 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-ff8cf4cc4-789d2_f3754bab-49a0-42b3-94f9-d875d57572ab/telemeter-client/0.log" Jul 22 16:55:50.331064 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:50.331039 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-ff8cf4cc4-789d2_f3754bab-49a0-42b3-94f9-d875d57572ab/reload/0.log" Jul 22 16:55:50.351637 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:50.351607 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-ff8cf4cc4-789d2_f3754bab-49a0-42b3-94f9-d875d57572ab/kube-rbac-proxy/0.log" Jul 22 16:55:51.944632 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:51.944594 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt"] Jul 22 16:55:51.945043 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:51.944951 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464bd6ee-320f-48d3-b167-5a2d356bcc10" containerName="istio-proxy" Jul 22 16:55:51.945043 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:51.944963 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="464bd6ee-320f-48d3-b167-5a2d356bcc10" containerName="istio-proxy" Jul 22 16:55:51.945043 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:51.944985 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13" containerName="revoke-keys" Jul 22 16:55:51.945043 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:51.944991 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13" containerName="revoke-keys" Jul 22 16:55:51.945043 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:51.945042 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a01d83a-b1ae-4f94-bedf-fd7e24cb2b13" containerName="revoke-keys" Jul 22 16:55:51.945202 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:51.945052 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="464bd6ee-320f-48d3-b167-5a2d356bcc10" containerName="istio-proxy" Jul 22 16:55:51.949579 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:51.949556 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:51.951963 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:51.951938 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bzfkk\"/\"kube-root-ca.crt\"" Jul 22 16:55:51.952082 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:51.951973 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bzfkk\"/\"default-dockercfg-s9whn\"" Jul 22 16:55:51.952587 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:51.952570 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bzfkk\"/\"openshift-service-ca.crt\"" Jul 22 16:55:51.962890 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:51.962861 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt"] Jul 22 16:55:52.065605 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.065570 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-lib-modules\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.065605 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.065609 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkfpb\" (UniqueName: \"kubernetes.io/projected/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-kube-api-access-pkfpb\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.065839 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.065641 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-proc\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.065839 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.065763 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-sys\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.065908 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.065847 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-podres\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.167331 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.167291 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-podres\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.167331 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.167338 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-lib-modules\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.167583 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.167357 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkfpb\" (UniqueName: \"kubernetes.io/projected/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-kube-api-access-pkfpb\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.167583 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.167393 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-proc\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.167583 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.167427 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-sys\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.167583 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.167483 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-podres\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.167583 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.167515 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-sys\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.167583 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.167516 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-proc\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.167583 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.167483 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-lib-modules\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.176107 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.176084 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkfpb\" (UniqueName: \"kubernetes.io/projected/5a2aa3f0-6acf-4480-af67-0d43f331b2eb-kube-api-access-pkfpb\") pod \"perf-node-gather-daemonset-qlcbt\" (UID: \"5a2aa3f0-6acf-4480-af67-0d43f331b2eb\") " pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.260748 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.260623 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:52.337906 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.337878 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/2.log" Jul 22 16:55:52.342415 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.342389 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-sf2mn_799e2b87-1954-4103-860d-33ab42170349/console-operator/3.log" Jul 22 16:55:52.395657 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.395631 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt"] Jul 22 16:55:52.398077 ip-10-0-131-21 kubenswrapper[2567]: W0722 16:55:52.398042 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5a2aa3f0_6acf_4480_af67_0d43f331b2eb.slice/crio-ad40d8dba65b81339bbf3a28ebf380508b1546e00f2fb42eb377f989f81fd60f WatchSource:0}: Error finding container ad40d8dba65b81339bbf3a28ebf380508b1546e00f2fb42eb377f989f81fd60f: Status 404 returned error can't find the container with id ad40d8dba65b81339bbf3a28ebf380508b1546e00f2fb42eb377f989f81fd60f Jul 22 16:55:52.461553 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:52.461523 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" event={"ID":"5a2aa3f0-6acf-4480-af67-0d43f331b2eb","Type":"ContainerStarted","Data":"ad40d8dba65b81339bbf3a28ebf380508b1546e00f2fb42eb377f989f81fd60f"} Jul 22 16:55:53.468394 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:53.468356 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" event={"ID":"5a2aa3f0-6acf-4480-af67-0d43f331b2eb","Type":"ContainerStarted","Data":"4042abf25bc95102eb0ccbb1d3203ebb91cd170dc7751e9cae71aab9870fb107"} Jul 22 16:55:53.468814 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:53.468521 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:55:53.489467 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:53.489417 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" podStartSLOduration=2.489400803 podStartE2EDuration="2.489400803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:55:53.486311491 +0000 UTC m=+2918.628185569" watchObservedRunningTime="2026-07-22 16:55:53.489400803 +0000 UTC m=+2918.631274882" Jul 22 16:55:53.517742 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:53.517715 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-ltpvx_dc08ab36-8b95-4b9f-a9dc-e3aade431059/volume-data-source-validator/0.log" Jul 22 16:55:54.522222 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:54.522191 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c78v9_db5fa8ab-0e89-4823-9f94-353bdf81e0dc/dns/0.log" Jul 22 16:55:54.541370 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:54.541344 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c78v9_db5fa8ab-0e89-4823-9f94-353bdf81e0dc/kube-rbac-proxy/0.log" Jul 22 16:55:54.695100 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:54.695064 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j8z9t_808dd442-3437-4347-a145-3cf2f7315f6c/dns-node-resolver/0.log" Jul 22 16:55:55.338898 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:55.338839 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xspks_d0d50a09-e266-4478-8e93-19b3aea03d79/node-ca/0.log" Jul 22 16:55:56.258909 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:56.258857 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678ndvrh_85f1c726-74bd-42a4-8f0d-342ac1e83aad/istio-proxy/0.log" Jul 22 16:55:56.387251 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:56.387179 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-8d5b7_252f357d-2451-4e46-9887-075ccae6dd38/discovery/0.log" Jul 22 16:55:56.572959 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:56.572904 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-586bddb798-57m2m_57dbd4ff-f4ed-484a-801b-9cef8a961409/router/0.log" Jul 22 16:55:57.312132 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:57.312090 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vprx2_bec7900e-a6b5-4e09-b908-690e4d13b982/serve-healthcheck-canary/0.log" Jul 22 16:55:57.811705 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:57.811670 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-24cbv_0f6ccc41-e229-40a9-9a3a-a6fa628dfe86/insights-operator/0.log" Jul 22 16:55:57.812232 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:57.812214 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-24cbv_0f6ccc41-e229-40a9-9a3a-a6fa628dfe86/insights-operator/1.log" Jul 22 16:55:57.833595 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:57.833573 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5bbfb_51215a2f-bce7-4a3d-a206-d96885c4c724/kube-rbac-proxy/0.log" Jul 22 16:55:57.854475 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:57.854438 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5bbfb_51215a2f-bce7-4a3d-a206-d96885c4c724/exporter/0.log" Jul 22 16:55:57.875869 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:57.875844 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5bbfb_51215a2f-bce7-4a3d-a206-d96885c4c724/extractor/0.log" Jul 22 16:55:59.484232 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:55:59.484192 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bzfkk/perf-node-gather-daemonset-qlcbt" Jul 22 16:56:00.952251 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:00.952223 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-587bf8b54-qg9rw_239e7a15-399f-4a8a-8341-edb28cf3faa4/manager/0.log" Jul 22 16:56:00.989634 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:00.989610 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-587bf8b54-qg9rw_239e7a15-399f-4a8a-8341-edb28cf3faa4/copy-manifests/0.log" Jul 22 16:56:02.063939 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:02.063912 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678ndvrh_85f1c726-74bd-42a4-8f0d-342ac1e83aad/istio-proxy/0.log" Jul 22 16:56:02.196213 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:02.196183 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-8d5b7_252f357d-2451-4e46-9887-075ccae6dd38/discovery/0.log" Jul 22 16:56:02.335985 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:02.335946 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-586bddb798-57m2m_57dbd4ff-f4ed-484a-801b-9cef8a961409/router/0.log" Jul 22 16:56:02.403765 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:02.403730 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-b9d69d95c-tlvgs_858a6d87-46bd-43b9-8056-3b05c72113c6/manager/0.log" Jul 22 16:56:02.430488 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:02.430459 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-9dcnv_ff65214b-edb6-4ea4-abf1-c690520ae97f/openshift-lws-operator/0.log" Jul 22 16:56:03.500456 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:03.500430 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678ndvrh_85f1c726-74bd-42a4-8f0d-342ac1e83aad/istio-proxy/0.log" Jul 22 16:56:03.618026 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:03.617994 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-8d5b7_252f357d-2451-4e46-9887-075ccae6dd38/discovery/0.log" Jul 22 16:56:03.767247 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:03.767167 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-586bddb798-57m2m_57dbd4ff-f4ed-484a-801b-9cef8a961409/router/0.log" Jul 22 16:56:07.740510 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:07.740473 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-6s889_957c1e60-c08f-437f-afaf-d3679b2e8376/kube-storage-version-migrator-operator/1.log" Jul 22 16:56:07.741275 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:07.741256 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-6s889_957c1e60-c08f-437f-afaf-d3679b2e8376/kube-storage-version-migrator-operator/0.log" Jul 22 16:56:08.897600 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:08.897571 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xq6wn_6052e9bc-e447-449b-86eb-d4484c381c9c/kube-multus-additional-cni-plugins/0.log" Jul 22 16:56:08.919111 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:08.919085 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xq6wn_6052e9bc-e447-449b-86eb-d4484c381c9c/egress-router-binary-copy/0.log" Jul 22 16:56:08.943651 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:08.943619 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xq6wn_6052e9bc-e447-449b-86eb-d4484c381c9c/cni-plugins/0.log" Jul 22 16:56:08.965141 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:08.965113 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xq6wn_6052e9bc-e447-449b-86eb-d4484c381c9c/bond-cni-plugin/0.log" Jul 22 16:56:08.987264 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:08.987239 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xq6wn_6052e9bc-e447-449b-86eb-d4484c381c9c/routeoverride-cni/0.log" Jul 22 16:56:09.008488 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:09.008459 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xq6wn_6052e9bc-e447-449b-86eb-d4484c381c9c/whereabouts-cni-bincopy/0.log" Jul 22 16:56:09.027925 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:09.027895 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xq6wn_6052e9bc-e447-449b-86eb-d4484c381c9c/whereabouts-cni/0.log" Jul 22 16:56:09.224600 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:09.224526 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pr97d_df7d36f9-eea8-4389-9631-d91bd24acc47/kube-multus/0.log" Jul 22 16:56:09.285806 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:09.285778 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6vr2t_8c507486-5095-4e26-ba26-9ac77c17590d/network-metrics-daemon/0.log" Jul 22 16:56:09.305788 ip-10-0-131-21 kubenswrapper[2567]: I0722 16:56:09.305756 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6vr2t_8c507486-5095-4e26-ba26-9ac77c17590d/kube-rbac-proxy/0.log"